Issue341

Title Generator objects and generator functions do not always have same code object flags
Priority bug Status resolved
Superseder Nosy List fussel, kayhayen
Assigned To kayhayen Keywords wrong_execution

Created on 2016-11-17.16:13:19 by fussel, last changed by kayhayen.

Files
File name Uploaded Type Edit Remove
nuitka_ws_purepy.zip fussel, 2016-11-28.10:41:52 application/zip
Messages
msg2494 (view) Author: kayhayen Date: 2018-08-23.23:10:53
This has been implemented dubbed "uncompiled generator integration" a few 
release ago, and so far covers 3.5 or higher in stable, 3.4 in develop.

This should solve it.

Yours,
Kay
msg2369 (view) Author: kayhayen Date: 2018-04-01.09:51:30
I did remove this recently. I kind of forgot about this issue. Not sure how, but 
maybe the tp_iter could be made to detect the YIELD_FROM case and behave 
differently.

Being iterable generally causes other issues.
msg2095 (view) Author: kayhayen Date: 2016-12-04.10:42:12
This is pretty shitty, but without tp_iter, the byte code GET_YIELD_FROM_ITER won't work. 
It type checks for uncompiled coroutines and then uncompiled generators, and then it must 
be iterable, which coroutines normally are not...

I guess, we will have to live with being incompatible there. Like I said, functional code 
will work. Few people really rely on coroutine objects not to be iterable with "iter". I 
will try and ask CPython to make it possible though to plug the system. 

Yours,
Kay
msg2094 (view) Author: kayhayen Date: 2016-12-03.19:22:28
The iterator interface is not it. This is from the CPython 3.5 test suite:

        async def foo():
            raise StopIteration

        check = lambda: self.assertRaisesRegex(
            TypeError, "'.*coroutine' object is not iterable")

        with check():
            list(foo())

Basically that means, that it must not be iterable, yet clearly, "yield from" in a co-routine does 
it or it is at least supposed to be able to do it as your example shows, and of course, how else 
would that work.

I need to get back to the drawing board here. This very likely doesn't affect correct code. I need 
to find out, how "yield from" achieves its goal without "tp_iter". I hope it doesn't type check too 
badly.

Yours,
Kay
msg2093 (view) Author: kayhayen Date: 2016-12-03.06:19:46
I pushed something to factory, which adds the iteration interface for coroutines 
too. I never saw it used when doing CPython tests, but that is likely because 
shortcuts being used all the time.

This makes it possible to mix compiled and uncompiled coroutine code freely I 
think.

Please find it on factory, and soon in a hotfix near you, likely 0.5.24.3 once 
it exists.

Yours,
Kay
msg2090 (view) Author: fussel Date: 2016-12-02.09:17:04
Hello Kay,

just tested your fix: if I build with --recurse-to=websockets the error is gone.
But the problem remains if I dont include the websockets package to the build.

I am on commit 	c10ca12803031ecc72ce19e317d3f94c662fbccf of your factory branch.

Regards,

Martin
msg2088 (view) Author: kayhayen Date: 2016-11-29.18:23:32
I just pushed a hotfix to factory. Thanks to your reproducer, I am pretty sure 
this was the original issue.

I invite you to test it, or if you so wish to wait for the hotfix to be made, I 
am only waiting for the buildbots to complete their checks.

Yours,
Kay
msg2087 (view) Author: kayhayen Date: 2016-11-29.11:21:12
So, this is the reason this happens.

a) The "types.coroutine" likes to replace code objects for generator functions with an extra 
indication 0x100 for co_flags.

b) Nuitka achieves the similar thing by monkey patching "types.coroutine" to change the code object 
of the function, directly modifying it.

c) The function that creates the generator object, and the generator object are supposed to use the 
same code object, and share that modification.

d) The way Nuitka achieves the sharing of the code object is by having only one per parameters. 
Creator function and generator objects must create the same code object for this to work.

e) For the generator down there in web sockets, a difference comes into play, that makes for 2 
different code objects (non-optimal, and of course a bug, one of them is wrong), and these are both 
being used.

f) "The enhanced types.coroutine" (which async.coroutine calls) changes only the one used by the 
generator creating function, the compiled generator object in this case, is then NOT getting that 
flag changed, and when it tries to yield from a coroutine, that kicks of the error handling.

So, these are the two code objects:

    codeobj_3c672f0360928c457f2d5685006d6e51 = MAKE_CODEOBJ( module_filename_obj, 
const_str_plain_handler, 55, const_tuple_str_plain_self_tuple, 1, 0, CO_GENERATOR | CO_OPTIMIZED | 
CO_NEWLOCALS | CO_NOFREE );

    codeobj_c258dbc450118fc62ef8539a7e2eb739 = MAKE_CODEOBJ( module_filename_obj, 
const_str_plain_handler, 55, const_tuple_str_plain_self_tuple, 1, 0, CO_GENERATOR | CO_OPTIMIZED | 
CO_NEWLOCALS );

The first one is used for the function, the later one for the generator. The CO_NOFREE is supposed 
to be set for functions that use no closure. The creating function, which would be the one that 
passes them along claims it doesn't have them, but the created generator seems to have them.

Hacking code objects for generators to always set the flag, your example was working. So the issue 
can be re-titled to a more matching description.
msg2086 (view) Author: kayhayen Date: 2016-11-29.07:14:08
Hello Martin,

thanks for providing this, very helpful!

So I am getting this:

Traceback (most recent call last):
  File "/data/home/hayen/repos/Py2C/t/main.dist/websockets/server.py", line 81, in handler
TypeError: cannot 'yield from' a coroutine object in a non-coroutine generator

I have yet to understand where that comes from, but I have a suspect. It seems to call this 
thing:

<bound compiled_method Server.coro_ws_poll of <__main__.Server object at 0x7f0ca40fed30>> 
<class 'compiled_method'>

So that is your code:

    async def coro_ws_poll(self, websocket, path):

And that's probably producing a compiled coroutine object. The other day, I added to the
factory branch something like this in generator's YIELD_FROM helper:

#if PYTHON_VERSION >= 350
        else if ( PyCoro_CheckExact( value ) )
        {
            retval = PyGen_Send( (PyGenObject *)value, Py_None );
        }
#endif

This mimics special code already there for normal generator objects. And it deals with the 
uncompiled co-routine objects only. For compiled co-routines, it seems that the equivalent 
is missing.

That of course seems pretty fixable. Hang on. :)

Yours,
Kay
msg2085 (view) Author: fussel Date: 2016-11-28.10:41:52
Hello Kay,

I have made an other example, very similar to the "Getting started" example of
the websockets doc: https://websockets.readthedocs.io/en/stable/intro.html.

In the nuitka_ws_purepy.zip are two files: the main.py, which is the websocket
server, and the wsclient.py, which is a pure python client. 
So there is no browser dependency at all. Launch the main.py first, either
compiled with nuitka or uncompiled. Then start python wsclient.py from a shell
(no need to compile).

The server and client will ping pong a little bit. The uncompiled code will
print some messages in the shell, thats it. 
The compiled main.py will fail with the error described in this issue
("'compiled_coroutine' object is not iterable") as soon as the wsclient.py connects.
The wsclient.py will then fail the aswell because the socket is closed by the
server, which is ok.

I hope this example is more reproducible.

Regards,

Martin
msg2084 (view) Author: kayhayen Date: 2016-11-26.07:16:22
Using the example you gave, first on Linux, and now on Windows, it just says 
"Invalid Request" when I query localhost:12345. There must be something wrong with 
the example. I did pip install websockets if that is the proper thing. I am sorry, 
can you make it more reproducible to me? I didn't find anything to enable logging 
that helped.
msg2079 (view) Author: fussel Date: 2016-11-22.08:57:15
Hello Kay,

I have attached a ZIP file including a main.py and an index.html. First I run
the main.py and then I load the index.html, e.g. with Chrome. As soon as the
browser loads the page, the Python console should print "browser hello". This
works fine as long I run it with the python interpreter.

The compiled application fails as the index.html is loaded and the browser
connects the Websocket. Now we get the "'compiled_coroutine' is not iterable" Error.

Best Regards,

Martin
msg2078 (view) Author: kayhayen Date: 2016-11-22.07:54:33
This is lacking a calling main, likely that is the difference, but it works for 
me just like that. Can you correct the Mini8.py attached to the bug report, so 
it more closely matches what you do? I assume you are calling it in another way, 
are you?

Yours,
Kay
msg2072 (view) Author: fussel Date: 2016-11-17.16:16:04
I am using nuitka 0.5.25rc2 and below
msg2071 (view) Author: fussel Date: 2016-11-17.16:13:18
Following code will compile, but fail on execution (once the browser connects to
the websocket). I am on py3.5.2, win7 x64.

import websockets

class Foo():
    
    async def setup(self):

        # serve() requires a coro as connection handler
        # this works fine as uncompiled code
        wsserver = await websockets.serve(self.coro_ws_poll, "127.0.0.1", 1234)

    async def coro_ws_poll(self, websocket, path):
        msg = await websocket.recv()
        print(str(msg))   


Trace is:

Error in connection handler
Traceback (most recent call last):
  File
"C:\eigene\entwicklung\app\frameworks\python\python35_x64\lib\site-packages\websockets\server.py",
line 81, in handler
    yield from self.ws_handler(self, path)
TypeError: 'compiled_coroutine' object is not iterable
Error in connection handler
Traceback (most recent call last):
  File
"C:\eigene\entwicklung\app\frameworks\python\python35_x64\lib\site-packages\websockets\server.py",
line 81, in handler
    yield from self.ws_handler(self, path)
TypeError: 'compiled_coroutine' object is not iterable
History
Date User Action Args
2018-08-23 23:10:53kayhayensetstatus: testing -> resolved
messages: + msg2494
2018-04-01 09:51:30kayhayensetmessages: + msg2369
2016-12-04 10:42:13kayhayensetmessages: + msg2095
2016-12-03 19:22:28kayhayensetmessages: + msg2094
2016-12-03 06:20:12kayhayensetfiles: - nuitka_ws.zip
2016-12-03 06:20:08kayhayensetfiles: - Mini8.py
2016-12-03 06:19:47kayhayensetmessages: + msg2093
2016-12-02 09:17:04fusselsetmessages: + msg2090
2016-11-29 18:23:32kayhayensetstatus: chatting -> testing
messages: + msg2088
2016-11-29 11:21:12kayhayensetmessages: + msg2087
title: compiled.exe fails on websocketserver calling a compiled coroutine -> Generator objects and generator functions do not always have same code object flags
2016-11-29 07:14:08kayhayensetmessages: + msg2086
2016-11-28 10:41:53fusselsetfiles: + nuitka_ws_purepy.zip
messages: + msg2085
2016-11-26 07:16:22kayhayensetmessages: + msg2084
2016-11-22 08:57:15fusselsetfiles: + nuitka_ws.zip
messages: + msg2079
2016-11-22 07:54:34kayhayensetfiles: + Mini8.py
assignedto: kayhayen
messages: + msg2078
keyword: + wrong_execution
nosy: + kayhayen
2016-11-17 16:16:04fusselsetstatus: unread -> chatting
messages: + msg2072
2016-11-17 16:13:19fusselcreate