Issue404

Title asyncio issue - RuntimeError: cannot reuse already awaited coroutine
Priority bug Status testing
Superseder Nosy List jflevesque, kayhayen
Assigned To kayhayen Keywords wrong_execution

Created on 2018-06-26.21:39:01 by jflevesque, last changed by jflevesque.

Messages
msg2438 (view) Author: jflevesque Date: 2018-07-20.19:33:35
Hi Kay,

I am using 0.5.31 on Python 3.6.6 for a few days and I am catching new exceptions 
that were catched as RuntimeError in the past. In this case, it's SSL context errors 
with aiohttp because Network is unreachable. Valid exceptions and now I see what is 
happening.

We are making progress for sure.

Thanks!
JF
msg2437 (view) Author: kayhayen Date: 2018-07-20.19:28:47
I had narrowed the error to occur for a non-recursing mere acceleration 
compilation, for which it is fixed. For standalone it seems another, similar 
issue is raised. I got to check, I think I noticed a couple of places that need 
to use that code, but it might not be complete.

This could be a bit hard to drill down to an actual minimal test case, might 
take more time. At least it is somewhat better for some cases already. Maybe it 
is more effective to make tests that do all the things, but that is a lot work 
too.

Yours,
Kay
msg2436 (view) Author: jflevesque Date: 2018-07-20.13:49:31
Hi Kay,

I still have the issue with the factory build. I updated the Pipfile on Github (https://github.com/levesquejf/nuitka-asyncio-bug) 
to use the factory build. I still have the "RuntimeError: cannot reuse already awaited coroutine" exception when running ./run-
nuitka.sh. I don't see the CancelledError Exception.

Thanks,
JF
msg2435 (view) Author: kayhayen Date: 2018-07-20.10:48:50
Hello,

I think I have it, works for your code I think, and compiles and ought to work 
with 3.5 and 3.7 too:

http://nuitka.net/doc/factory.html

So this was quick in the end. I wanted to finish it up. But factory might be 
unstable due to other ongoing cleanups for class dict optimization. But let me 
know, if you agree your example works better now. Hope I didn't break it while 
making changes for other versions.

Yours,
Kay
msg2434 (view) Author: kayhayen Date: 2018-07-20.06:51:26
And yeah, I just got the correct behavior out of it. The "throw" happens in a 
lot of places, and seems the "yield from" of coroutines was the one place, where 
this happens.

But basically I had to copy private code from CPython a lot. But it is not good 
yet, will take me a while to properly get this integrated. For full 
compatibility it will be unavoidable, and I will have to check for 3.5, 3.7 
differences too.

Hang on, I am hoping to make a pre-release with this during the week.

Yours,
Kay
msg2432 (view) Author: kayhayen Date: 2018-07-19.17:42:55
Nah, wait, I was premature. The code in question is duplicated a lot in Nuitka, 
once for generators, once for coroutines, and once for asyncgen, so this is 
still on.

Yours,
Kay
msg2431 (view) Author: kayhayen Date: 2018-07-19.15:44:15
I just tried out my theory, and came to realize, that the code in question is not really the 
one that can cause the issue. No "throw" issue is going on.

Maybe other code parts need special handling there.

Seems I was stupid, and need to check harder for why this happens. There is not that many 
things going on. Unfortunately it is not clear, which coroutine has the RuntimeError. I will 
try and add even more traces for it and compare where the "async with" treats it 
differently. 

Maybe it would be possible to use an uncompiled async object instead, that traces what 
happens to it, but I don't know how.

Yours,
Kay
msg2426 (view) Author: jflevesque Date: 2018-07-13.21:33:28
These exceptions are really giving me a hard time so it’s great you plan to fix them in 
0.5.32. Do you have an idea when it will be released?

Thanks all your great work Kay!

Jean-Francois
msg2423 (view) Author: kayhayen Date: 2018-07-09.06:31:01
I fixed the cyclic import thing, something changed in 3.5 that so far had escaped my attention.

However, the "enum" module really hates compiled "__new__" methods. It calls them manually and the 
staticmethod is not callable, but we have to make that annotation, because otherwise it's an issue, with only 
non-compiled functions being added behind our back.

Not sure what to do there. 

I am going to attack this issue proper for the 0.5.32 release cycle, adding the duplication of the CPython 
code that is needed. Recursing into stdlib is of course not very important, but your issue is.

I just made the overdue 0.5.31 without a fix for this, sorry.

Yours,
Kay
msg2422 (view) Author: kayhayen Date: 2018-07-06.09:24:47
So in trying the suggested workaround, I found a regression in at least the factory version, and a not 
yet supported feature for Python3.

Seems that cyclic imports of "multiprocessing" show that from x import y not only imports "x" and then 
does an attribute lookup on "x", but if that fails, it also does the actual import, which is kind of 
surprising. I am fixed this right now.

This highlights that for Python3 I probably never so far did use --recurse-stdlib. Obviously cyclic 
dependencies of that kind are pretty rare in actual code, esp. as Python2 never worked with that.

So, the next pre-release ought to work for you with that option then.

Yours,
Kay
msg2421 (view) Author: kayhayen Date: 2018-07-05.11:33:31
So this is an issue with lack of support for uncompiled coroutines, or so I now suspect.

Nuitka closes "yield from" values in the fallback way, i.e. to call "throw()" on the "yielded from" value, then does unwrap the value of that from the 
exception returned if possible, and resumes locally.

However, I noticed that there is suspect code in gen_throw, which is executed for hard type checks on this, in CPython:

            gen->gi_frame->f_lasti += sizeof(_Py_CODEUNIT);
            if (_PyGen_FetchStopIterationValue(&val) == 0) {
                ret = gen_send_ex(gen, val, 0, 0);
                Py_DECREF(val);
            } else {
                ret = gen_send_ex(gen, Py_None, 1, 0);
            }

Notice how it moves the instruction pointer before resuming the generator. Amazing grace, this is "creative" for a lack of better term.

My theory is that without this trick, it will repeat what it did last, i.e. attempting to continue to yield from, then causing the error to happen.

Effectively that will mean to reproduce that code in Nuitka, so it can emulate the same piece of garbage this is. I need to check how old that specific 
trick is, and try to get a more minimal reproducer too.

That won't be quick I fear.

The only weak point is my argument is that standalone mode ought to not expose the same issue. But since "asyncio" is in standard library, it is not 
compiled by default, but included as bytecode. Therefore I am trying with --recurse-stdlib now, and if that works, it would be a workaround. However, 
that is growing compile time quiet a bit I fear. Maybe limiting to "asyncio" would be wiser, but the command line currently doesn't allow for that.

Yours,
Kay
msg2420 (view) Author: kayhayen Date: 2018-07-05.10:08:20
So I have added traces to show why Nuitka gives this error in its compiled coroutine implementation, 
but turns out, it's a non-compiled coroutine that does it, which makes sense, as it's the aiohttp one 
that is being active at the time, but inside an compiled async, likely through a wrapper object.

So, I am now assuming, that this is about how a thrown GeneratorExit into the compiled generator ends 
up being forwarded to the async context manager.

The funny thing is how in the non-standalone case, Nuitka compiled coroutine code is really only 
operating non-compiled coroutines, which should also make it easier to isolate. I will need to add more 
traces to see what's going on in the cancel case.

Yours,
Kay
msg2419 (view) Author: kayhayen Date: 2018-07-05.09:45:07
Just to note, standalone mode makes no difference.

I looked at the code of Python-3.6.5/Modules/_asynciomodule.c and it seems to 
not make any far reaching assumptions about what it is awaiting. That should not 
be it.

Yours,
Kay
msg2418 (view) Author: kayhayen Date: 2018-07-05.09:14:46
Hello there,

sorry, not yet. I have been focusing on 3.7 support recently, but given the quality of your 
report, it's a shame, and I had a quick look just now.

The cancel of a task in asyncio seems to delegate to a "fut_waiter" cancel, that I have not 
yet identified the code of. I need to check that out. I didn't see anything obvious there, 
but I am assuming that some kind of check is the coroutine is finished, is not compatible.

These being "async def" as opposed to decorator declared coroutines, I have no immediate 
idea what could be wrong there.

I will fire up a throughaway virtualenv, and trace things changing asyncio code. I also 
wondered if the 10 second running couldn't be achieved with a mere async sleep for less code 
surface. Unfortunately these coroutine things have thee weirdest control flows and are 
always hard to debug. I will keep you posted however.

Yours,
Kay
msg2417 (view) Author: jflevesque Date: 2018-07-05.01:42:02
Have you been able to reproduce on your end and to investigate the issue? Do you think 
it will be easy to fix? What is your approximate timeline for a fix? Thanks!
msg2416 (view) Author: kayhayen Date: 2018-06-27.05:36:04
Thanks for the great report. Looks like a bug to me. The functionality is 
quiet involved, and bugs in Nuitka are more than likely. But don't despair, 
with a good reproducer like that I might be able to fix it quickly.

Yours,
Kay
msg2415 (view) Author: jflevesque Date: 2018-06-26.21:38:59
Good afternoon!

I am having issues with asyncio and Nuitka. Once in a while (mostly when something bad happens), I get some "RuntimeError: cannot reuse already awaited coroutine" exceptions from my Nuitka compiled code. This never happens with "native" CPython.

I have been able to reproduce the issue (Nuitka 0.5.30, Python 3.6.5, all running on Linux). In my example, with python, the exception is CancelledError.

The code used to reproduce and the details of the output are available at: https://github.com/levesquejf/nuitka-asyncio-bug

Is this a bug? Something bad in my asyncio usage?

Thanks,

Jean-Francois
History
Date User Action Args
2018-07-20 19:33:35jflevesquesetmessages: + msg2438
2018-07-20 19:28:47kayhayensetmessages: + msg2437
2018-07-20 13:49:31jflevesquesetmessages: + msg2436
2018-07-20 10:55:20kayhayensetstatus: in-progress -> testing
2018-07-20 10:48:50kayhayensetmessages: + msg2435
2018-07-20 06:51:27kayhayensetmessages: + msg2434
2018-07-19 17:42:55kayhayensetmessages: + msg2432
2018-07-19 15:44:15kayhayensetmessages: + msg2431
2018-07-13 21:33:28jflevesquesetmessages: + msg2426
2018-07-09 06:31:01kayhayensetmessages: + msg2423
2018-07-06 09:24:47kayhayensetmessages: + msg2422
2018-07-05 11:33:31kayhayensetmessages: + msg2421
2018-07-05 10:08:20kayhayensetmessages: + msg2420
2018-07-05 09:45:07kayhayensetmessages: + msg2419
2018-07-05 09:14:47kayhayensetmessages: + msg2418
2018-07-05 01:42:03jflevesquesetmessages: + msg2417
2018-06-27 05:36:15kayhayensetassignedto: kayhayen
nosy: + kayhayen
2018-06-27 05:36:04kayhayensetstatus: unread -> in-progress
messages: + msg2416
keyword: + wrong_execution
2018-06-26 21:39:01jflevesquecreate