Issue399

Title Issues with Multiprocessing: ModuleNotFoundError
Priority bug Status testing
Superseder Nosy List Oni, kayhayen
Assigned To kayhayen Keywords

Created on 2018-04-24.18:03:19 by Oni, last changed by Oni.

Files
File name Uploaded Type Edit Remove
multiprocessing.zip Oni, 2018-04-24.18:03:19 application/x-zip-compressed
Messages
msg2410 (view) Author: Oni Date: 2018-06-01.15:35:25
Now this bug seems related to: http://bugs.nuitka.net/issue371
msg2396 (view) Author: Oni Date: 2018-05-07.08:07:56
Update: still fails for me.

nuitka3 --version
0.5.30

nuitka3 --recurse-all --recurse-to=multiprocessing --plugin-enable=multiprocessing my_foo.py --output-dir=bin/
Nuitka:WARNING:C:\Foo\foo:1: Cannot find '_frozen_importlib_external' in package 'foo' as absolute import (tried 
_frozen_importlib_external).
Nuitka:INFO:Injecting plug-in based post load code for module 'multiprocessing.reduction':
Nuitka:INFO:    Monkey patching "multiprocessing" for compiled methods.
Nuitka:INFO:Injecting plug-in based pre load code for module 'multiprocessing.spawn':
Nuitka:INFO:    Monkey patching "multiprocessing" load environment.

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
File "C:\Miniconda3\envs\nuitka_test\lib\site-packages\nuitka\build\SingleExe.scons", line 396, in createEnvironment

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
File "C:\Miniconda3\envs\nuitka_test\lib\site-packages\nuitka\build\SingleExe.scons", line 1658, in <module>

If I try to run the executable:
PS C:\Foo\bin> ./my_foo.exe
Traceback (most recent call last):
  File "C:\Foo\my_foo.py", line 1, in <module>
    from multiprocessing import freeze_support
  File "C:\Miniconda3\envs\nuitka_test\lib\multiprocessing\__init__.py", line 16, in <module multiprocessing>
    from . import context
  File "C:\Miniconda3\envs\nuitka_test\lib\multiprocessing\context.py", line 6, in <module multiprocessing.context>
    from . import reduction
  File "C:\Miniconda3\envs\nuitka_test\lib\multiprocessing\reduction.py", line 19, in <module 
multiprocessing.reduction>
    from . import context
ImportError: cannot import name 'context'
msg2395 (view) Author: Oni Date: 2018-04-26.13:59:07
Thank you!
msg2394 (view) Author: kayhayen Date: 2018-04-26.13:26:33
Good news, a solution is coming in the next release, which again is coming very 
soon.
msg2393 (view) Author: Oni Date: 2018-04-26.10:08:41
I've installed factory nuitka but the situation didn't change much:

nuitka3 --recurse-all --plugin-enable=multiprocessing my_foo.py --output-dir=bin/
Nuitka:WARNING:C:\Foo\foo:1: Cannot find '_frozen_importlib_external' in package 'foo' as 
absolute import (tried _frozen_importlib_external).

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not 
set correctly
File "C:\python36\lib\site-packages\nuitka\build\SingleExe.scons", line 396, in 
createEnvironment

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not 
set correctly
File "C:\python36\lib\site-packages\nuitka\build\SingleExe.scons", line 1658, in <module>

Note 1:
the warning 'Nuitka:WARNING:foo/__main__.py:8: Cannot find 'multiprocessing.forking' as 
absolute import.' is gone this time!
Note 2:
the warning 'Nuitka:WARNING:C:\Foo\foo:1: Cannot find '_frozen_importlib_external' in package 
'foo' as absolute import (tried _frozen_importlib_external).' is always there, even if I 
compile a script without multiprocessing.

If I try to run the executable:
PS C:\Foo\bin> ./my_foo.exe
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "c:\python36\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
ModuleNotFoundError: No module named 'foo'
        < interrupt here with ^C >
Traceback (most recent call last):
  File "C:\Foo\my_foo.py", line 8, in <module>
    entry.main()
  File "C:\Foo\foo\entry.py", line 23, in main
    print(server_channel.recv())
  File "C:\Python36\Lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "C:\Python36\Lib\multiprocessing\connection.py", line 306, in _recv_bytes
    [ov.event], False, INFINITE)
KeyboardInterrupt
msg2392 (view) Author: Oni Date: 2018-04-26.08:34:04
Quick note:
"Visual Studio Build Tools" is just the compiler for C/C++ etc., linker etc. 
for building stuff in windows *without* the Visual Studio IDE itself.
It's sort of like installing gcc on linux.

As far as I know, "Visual Studio Build Tools" powers both, "Visual Studio 
Community 2017" (free-as-in-beer IDE) and the standard "Visual Studio" non-
free IDE.

I'll do some tests myself on factory. Thank you Kay!
msg2391 (view) Author: kayhayen Date: 2018-04-25.08:53:57
So I cherry-picked from there, and it mostly works. Still need to get feedback 
why it is still considered WIP.

I pushed it to factory: http://nuitka.net/doc/factory.html

Also, I am going to add your test case, so this is tested. It didn't work with 
Python2, likely relative/absolute import issue, but that's easy to fix.

Yours,
Kay
msg2390 (view) Author: kayhayen Date: 2018-04-25.08:33:13
This is a pull request on Github that declared itself WIP, but looks as if it 
does some of things at least: https://github.com/kayhayen/Nuitka/pull/57

I asked about the status, the Nuitka it based on might be a bit stale, and not 
work for other reasons. I might cherry-pick what is there and see what it does 
for us.

Yours,
Kay
msg2389 (view) Author: kayhayen Date: 2018-04-25.08:28:05
Hello there,

I think that multiprocessing on Python3.6 has had some changes, that nobody 
looked into yet.

Also curious, VSBuildtools 2017, is that from the non-free-in-beer version of VS 
2017?

Yours,
Kay
msg2386 (view) Author: Oni Date: 2018-04-24.18:03:19
Hi all,
I have an issue when using Nuitka to compile my app with multiprocessing module.

My app has a complex structure, so I created a minimal not-funtioning app (attached) to help pinpoint the 
issue.

Probably I don't know how to deal with multiprocessing and Nuitka, but I'm short of ideas so I'm opening this 
issue.

The minimal app, like my larger app, can be run directly:
python my_foo.py
or as a module:
python -m foo

If I try to compile with nuitka, I get:
nuitka3 --recurse-all --plugin-enable=multiprocessing my_foo.py --output-dir=bin/
Nuitka:WARNING:C:\Foo\foo:1: Cannot find '_frozen_importlib_external' in package 'foo' as absolute import 
(tried _frozen_importlib_external).
Nuitka:WARNING:my_foo.py:13: Cannot find 'multiprocessing.forking' as absolute import.

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
File "C:\python36\lib\site-packages\nuitka\build\SingleExe.scons", line 396, in createEnvironment

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
File "C:\python36\lib\site-packages\nuitka\build\SingleExe.scons", line 1651, in <module>

I'm using VSBuildtools 2017 and I'm pretty sure they are causing the scons warnings. Anyway the executable is 
produced and if I try to run it, I get:

PS C:\Foo\bin> ./my_foo.exe
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "c:\python36\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
ModuleNotFoundError: No module named 'foo'

I've guessed that perhaps multiprocessing is spawning a process that gets lost in the module system, so I've 
tried:
nuitka3 --recurse-all --plugin-enable=multiprocessing foo/__main__.py --output-dir=bin/
Nuitka:WARNING:C:\Foo\foo:1: Cannot find '_frozen_importlib_external' in package 'foo' as absolute import 
(tried _frozen_importlib_external).
Nuitka:WARNING:foo/__main__.py:8: Cannot find 'multiprocessing.forking' as absolute import.

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
File "C:\python36\lib\site-packages\nuitka\build\SingleExe.scons", line 396, in createEnvironment

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
File "C:\python36\lib\site-packages\nuitka\build\SingleExe.scons", line 1651, in <module>

But even in this case the executable fails:
PS C:\Foo\bin> ./__main__.exe
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "c:\python36\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "c:\python36\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "c:\python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "c:\python36\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "c:\python36\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "c:\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Foo\foo\__main__.py", line 1, in <module>
    from foo import entry
ModuleNotFoundError: No module named 'foo'
Traceback (most recent call last):
  File "C:\Foo\foo\__main__.py", line 5, in <module>
    entry.main()
  File "C:\Foo\foo\entry.py", line 23, in main
    print(server_channel.recv())
  File "C:\Python36\Lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "C:\Python36\Lib\multiprocessing\connection.py", line 306, in _recv_bytes
    [ov.event], False, INFINITE)
KeyboardInterrupt


My setup:

nuitka3 --version
0.5.29.4
with
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 16:07:46) [MSC v.1900 32 bit (Intel)] on win32
and with VSTools 2017 installed.

OS: Win10 64bit
History
Date User Action Args
2018-06-01 15:35:25Onisetmessages: + msg2410
2018-05-07 08:07:58Onisetmessages: + msg2396
2018-04-26 13:59:08Onisetmessages: + msg2395
2018-04-26 13:26:41kayhayensetstatus: chatting -> testing
2018-04-26 13:26:33kayhayensetassignedto: kayhayen
messages: + msg2394
nosy: + kayhayen
2018-04-26 10:08:41Onisetmessages: + msg2393
2018-04-26 08:34:05Onisetmessages: + msg2392
2018-04-25 08:53:57kayhayensetmessages: + msg2391
2018-04-25 08:33:13kayhayensetmessages: + msg2390
2018-04-25 08:28:05kayhayensetstatus: unread -> chatting
messages: + msg2389
2018-04-24 18:03:19Onicreate