Issue371

Title Multiprocessing ImportError on python >= 3.6.0
Priority bug Status chatting
Superseder Nosy List jeremy
Assigned To Keywords

Created on 2017-07-02.19:04:48 by jeremy, last changed by kayhayen.

Files
File name Uploaded Type Edit Remove
py3.6.0_verbose_debug_trace.txt jeremy, 2017-07-03.20:37:47 text/plain
Messages
msg2176 (view) Author: kayhayen Date: 2017-07-09.17:54:04
The difference there seems to be AbstractReducer which has code like this:

+    def __init__(self, *args):
+        register(type(_C().f), _reduce_method)
+        register(type(list.append), _reduce_method_descriptor)
+        register(type(int.__add__), _reduce_method_descriptor)
+        register(functools.partial, _reduce_partial)
+        register(socket.socket, _reduce_socket)

I kind of suspect, that this just doesn't work for compiled methods and needs 
some patching there. I will look into this the coming days. Not sure if I can 
find it before release, but of course that's important stuff to have 
multiprocessing work. Seems also Nuitka is missing its own test to cover that.

Yours,
Kay
msg2166 (view) Author: jeremy Date: 2017-07-03.20:37:47
Please disregard the last removed message, that was for an issue I encountered with 
array.so which I'm still troubleshooting. 

Below is test result for 3.6.0 with verbose, debug, and trace-execution enabled (nuitka -
-verbose --debug --trace-execution --portable --recurse-to=multiprocessing --python-
version=3.6 scratch.py && scratch.dist/scratch.exe). Attached is the full log. 

3.6.0 (truncated, please see attachment for full):
====================================
Traceback (most recent call last):
  File "/app/scratch.dist/scratch.py", line 1, in <module>
    import multiprocessing
  File "/app/scratch.dist/multiprocessing/__init__.py", line 16, in multiprocessing
  File "/app/scratch.dist/multiprocessing/context.py", line 6, in context
  File "/app/scratch.dist/multiprocessing/reduction.py", line 19, in reduction
ImportError: cannot import name 'context'
====================================
msg2164 (view) Author: kayhayen Date: 2017-07-03.13:13:29
You didn't try 3.6.0, did you?

Thanks,
Kay
msg2163 (view) Author: jeremy Date: 2017-07-02.19:04:48
Hi,

When importing multiprocessing on Python versions at or greater than 3.6.1, an 
ImportError is encountered. I haven't had a chance to look closely at the 
context/reduction modules for the imports. It's failing at a relative import, and to 
me is behaving as a cyclic import. I tested 3.5.3, 3.6.1, 3.6.2rc1. Within that set, 
the problem emerges at 3.6.1. To demonstrate:

scratch.py:
====================================
import multiprocessing
====================================

I had a docker harness handy, apologies for the verbosity. The command being executed 
for each container is below, where $V is either 3.5 or 3.6:

pip install nuitka && cd /app && nuitka --portable --recurse-to=multiprocessing --
python-version=$V scratch.py && scratch.dist/scratch.ex


3.6.1:
====================================
$ docker-compose run py3.6.1
Collecting nuitka
  Downloading Nuitka-0.5.26.tar.gz (1.6MB)
    100% |████████████████████████████████| 1.6MB 941kB/s
Building wheels for collected packages: nuitka
  Running setup.py bdist_wheel for nuitka ... done
  Stored in directory: 
/root/.cache/pip/wheels/16/cb/54/a8177c735befb8f26adbf41486c268d264ff3f6d22de1ab466
Successfully built nuitka
Installing collected packages: nuitka
Successfully installed nuitka-0.5.26
Traceback (most recent call last):
  File "/app/scratch.dist/scratch.py", line 1, in <module>
    import multiprocessing
  File "/app/scratch.dist/multiprocessing/__init__.py", line 16, in multiprocessing
  File "/app/scratch.dist/multiprocessing/context.py", line 6, in context
  File "/app/scratch.dist/multiprocessing/reduction.py", line 19, in reduction
ImportError: cannot import name 'context'
====================================


3.6.2rc1:
====================================
$ docker-compose run py3.6.2rc1
Collecting nuitka
  Downloading Nuitka-0.5.26.tar.gz (1.6MB)
    100% |████████████████████████████████| 1.6MB 898kB/s
Building wheels for collected packages: nuitka
  Running setup.py bdist_wheel for nuitka ... done
  Stored in directory: 
/root/.cache/pip/wheels/16/cb/54/a8177c735befb8f26adbf41486c268d264ff3f6d22de1ab466
Successfully built nuitka
Installing collected packages: nuitka
Successfully installed nuitka-0.5.26
Traceback (most recent call last):
  File "/app/scratch.dist/scratch.py", line 1, in <module>
    import multiprocessing
  File "/app/scratch.dist/multiprocessing/__init__.py", line 16, in multiprocessing
  File "/app/scratch.dist/multiprocessing/context.py", line 6, in context
  File "/app/scratch.dist/multiprocessing/reduction.py", line 19, in reduction
ImportError: cannot import name 'context'
====================================


Expected output/3.5.3:
====================================
$ docker-compose run py3.5.3
Collecting nuitka
  Downloading Nuitka-0.5.26.tar.gz (1.6MB)
    100% |████████████████████████████████| 1.6MB 1.0MB/s
Building wheels for collected packages: nuitka
  Running setup.py bdist_wheel for nuitka ... done
  Stored in directory: 
/root/.cache/pip/wheels/16/cb/54/a8177c735befb8f26adbf41486c268d264ff3f6d22de1ab466
Successfully built nuitka
Installing collected packages: nuitka
Successfully installed nuitka-0.5.26
====================================

Thanks,

Jeremy Blow
History
Date User Action Args
2017-07-09 17:54:04kayhayensetmessages: + msg2176
2017-07-03 21:23:24jeremysettitle: Multiprocessing ImportError on python >= 3.6.1 -> Multiprocessing ImportError on python >= 3.6.0
2017-07-03 20:37:47jeremysetfiles: + py3.6.0_verbose_debug_trace.txt
messages: + msg2166
2017-07-03 20:27:37jeremysetmessages: - msg2165
2017-07-03 20:27:11jeremysetmessages: + msg2165
2017-07-03 13:13:30kayhayensetstatus: unread -> chatting
messages: + msg2164
2017-07-02 19:04:48jeremycreate