Issue389

Title Crashing on compiled extension module when importing multiple times and then using types.coroutine
Priority bug Status resolved
Superseder Nosy List kayhayen, luis
Assigned To kayhayen Keywords

Created on 2017-12-01.08:14:51 by luis, last changed by kayhayen.

Files
File name Uploaded Type Edit Remove
import_crash.zip luis, 2017-12-01.08:14:49 application/x-zip-compressed
Messages
msg2367 (view) Author: kayhayen Date: 2018-03-31.14:37:43
Released as 0.5.29.3
msg2366 (view) Author: luis Date: 2018-03-30.16:44:42
Hi Kay,

Thanks a lot for the thorough analysis.

Apologies are not needed, I imagine it's a lot of effort from your side to 
maintain this project.

I'm glad to see that the fix is simple - once you know what the problem is, of 
course, that part doesn't look simple to me.

Thank you very much for the effort.

Best regards,

Luis.
msg2365 (view) Author: kayhayen Date: 2018-03-30.13:03:46
Correction to last note "Even in module mode some hacks are applied" it should say.

Patching twice overwrite the "old_types_coroutine" value, and then calling it, will lead to endless 
recursion in the C code, which apparently only leads to an abort on my system. I might have configured crash 
reports away, as CPython test suites crash with CPython way too often. 

That probably explains the different behavior.

The fix is a simple "if (!is_done) return" that shortcuts re-entry. I am pushing it to factory soon, but I 
also expect to make a hotfix out of it later today.

Thanks for the patience. Sorry for being unresponsive.

Yours,
Kay
msg2364 (view) Author: kayhayen Date: 2018-03-30.12:51:33
Ah yes, now I know.

In the module mode, not all hacks are applied, hope is for them to not be needed. It seems that the crash is happening in 
"types.coroutine", which Nuitka patches to use its own code, which shows more respect for compiled generators.

However, it seems to do that once per loaded module. That means, twice in your example, and then the patching does not work 
well with itself. Which is another, even worse issue, because it might have to.

So, fixing the patching to be a one time action for module mode solves the crash, but hides this other bug. But seems totally 
solvable.

Yours,
Kay
msg2362 (view) Author: kayhayen Date: 2018-03-30.12:28:37
So, when I recursed to tornado and used --trace-execution, it became apparent that the import of 
"asyncio" aborts heavily.

So instead of "tornado.web" importing "asyncio" from standard library already causes this to 
happen. Which is very bad news, but rings a bell. There have been similar reports.
msg2361 (view) Author: kayhayen Date: 2018-03-30.12:16:31
I noticed only now, this is not about standalone, but module mode. That is of 
course a different beast.

I managed to reproduce it, but no crash, just silent abort. The import of 
tornado.web is the last thing attempted. This happens with both MSVC and 
MinGW64.
msg2360 (view) Author: kayhayen Date: 2018-03-30.07:40:43
I was hoping for this to go away, after I made the test that finds this all across the board and 
fixing things found by it. Obviously that will only see modules that happen to be already installed 
on my system, which tornado was not. But often they use the same underlying trouble makers, e.g. 
crypto modules.

I am looking into this right now. Sorry for the delay. So far I can say, works for me on Linux 
(Debian). I will check on Windows too.

Yours,
Kay
msg2358 (view) Author: luis Date: 2018-03-29.09:31:06
Just to let you know, the problem is also present in 0.5.29.1 version (Python 
3.5 64 bits)
msg2302 (view) Author: luis Date: 2018-02-28.06:33:25
Hi again,

Two months now from the original post and no feedback :(

I was hoping to have some information here, since it is easy to reproduce 
with three lines code (included by the way) and it uses a very popular 
package like tornado.

To summarize, if I compile any code as module (pyd) that imports 
tornado.web, python crashes when you use the module. If you compile the same 
code as a stand alone exe, it works. Windows and Mingw, python 3.5, nuitka 
0.5.28.1.

I'd do really appreciate any feedback on this.

Thank you.

Luis.
msg2295 (view) Author: luis Date: 2017-12-01.08:14:49
Hi there,

I am using Nuitka on Windows to compile a package for my project. Inside that package I have a module that 
imports tornado.web. 

If I build a compiled package, Python crashes in the moment of importing tornado.web. But it doesn't crash if I 
compile an exe with the same package embedded.

And I mean crashes, not an exception. It crashes Python interpreter itself: "Python has stopped working"

I am attaching a very simple example to show how just the import makes it crash.

test.py
testmodule
|
 ----------- __init__.py
|
 ----------- testmodule.py

Where
 * __init__.py is empty
 * testmodule.py contains:
   
   from tornado import web

   print('Hello World')
 * test.py contains:
   from testmodule import testmodule
   
And that's it.

To compile only the package, I'm using:

nuitka --mingw --module testmodule --recurse-directory=testmodule --recurse-not-to=tornado --output-
dir=bin_module

If I go to bin_module and:

python
>>> form testmodule import testmodule -> CRASH


But if I compile test.py with the module embedded:

nuitka --mingw test.py --recurse-to=testmodule --recurse-not-to=tornado --output-dir=bin_exe

And if you run test.exe in bin_exe -> OK!! You have your 'Hello Horld'

I really need to import tornado.web in my compiled package. Any clue what is happening here?

Thank you!

Luis.
History
Date User Action Args
2018-03-31 14:37:43kayhayensetstatus: testing -> resolved
messages: + msg2367
2018-03-30 18:30:36kayhayensettitle: Python crashing on compiled module when importing tornado.web -> Crashing on compiled extension module when importing multiple times and then using types.coroutine
2018-03-30 16:44:43luissetmessages: + msg2366
2018-03-30 13:04:01kayhayensetassignedto: kayhayen
nosy: + kayhayen
2018-03-30 13:03:54kayhayensetstatus: chatting -> testing
2018-03-30 13:03:47kayhayensetmessages: + msg2365
2018-03-30 12:51:33kayhayensetmessages: + msg2364
2018-03-30 12:28:37kayhayensetmessages: + msg2362
2018-03-30 12:16:32kayhayensetmessages: + msg2361
2018-03-30 07:40:44kayhayensetmessages: + msg2360
2018-03-29 09:31:06luissetmessages: + msg2358
2018-03-13 18:38:50kayhayensetpriority: urgent -> bug
2018-02-28 06:33:28luissetpriority: bug -> urgent
status: unread -> chatting
messages: + msg2302
title: Python crashing on compiled module when imported tornado.web -> Python crashing on compiled module when importing tornado.web
2017-12-01 08:14:51luiscreate