Issue412

Title Nested namespace packages aren't included
Priority bug Status resolved
Superseder Nosy List kayhayen, morefigs
Assigned To kayhayen Keywords wrong_execution

Created on 2018-08-09.08:01:59 by morefigs, last changed by kayhayen.

Files
File name Uploaded Type Edit Remove
minimal-imports-example.zip morefigs, 2018-08-10.02:14:47 application/x-zip-compressed
Messages
msg2481 (view) Author: kayhayen Date: 2018-08-16.09:17:49
Part of 0.5.32.5 hotfix, thanks for the report.
msg2480 (view) Author: kayhayen Date: 2018-08-13.08:59:16
This is on factory, but kind of stuck there with goto generator changes, I will 
make it part of the next pre-release though.

Yours,
Kay
msg2479 (view) Author: morefigs Date: 2018-08-13.02:35:12
Thanks for looking into a fix, I look forward to it.
msg2478 (view) Author: kayhayen Date: 2018-08-10.14:14:51
I noticed for that spec based importing, no traces are given and added them. Seems we were asked:

import some_package.sub_package # considering responsibility
import some_package.sub_package # denied responsibility

and then

import some_package.sub_package.some_sub_module # considering responsibility
import some_package.sub_package.some_sub_module # claimed responsibility (compiled)

This is from --python-flag=-v which asks to explain. The traces are the ones given for pre-3.5, that 
will be given in the future too.

So how does that happen. Checking the module table gives:

static struct Nuitka_MetaPathBasedLoaderEntry meta_path_loader_entries[] =
{
    { (char *)"__main__", MOD_INIT_NAME( __main__ ), 0, 0, NUITKA_COMPILED_MODULE },
    { (char *)"some_package", MOD_INIT_NAME( some_package ), 0, 0, NUITKA_PACKAGE_FLAG },
    { (char *)"some_package.some_module", MOD_INIT_NAME( some_package$some_module ), 0, 0, 
NUITKA_COMPILED_MODULE },
    { (char *)"some_package.sub_package.some_sub_module", MOD_INIT_NAME( 
some_package$sub_package$some_sub_module ), 0, 0, NUITKA_COMPILED_MODULE },
    { (char *)"sub_package", MOD_INIT_NAME( sub_package ), 0, 0, NUITKA_PACKAGE_FLAG },
    { NULL, NULL, 0, 0, 0 }
};

As can be seen, the modules are OK, but sub_package is included without a package name. And that 
most probably is the downfall of it. Existing namespace packages can make up for it if they still 
exist, but if they do not, we never can find "some_package.sub_package.some_sub_module" which is 
what happens.

So this seems to be as simple as passing full package name over base module name. I am testing a fix 
internally, expect something relatively soon.

Yours,
Kay
msg2477 (view) Author: kayhayen Date: 2018-08-10.13:55:27
Well. seems that compiled code is used for as along as the namespace dirs exist

Also one initial suspect was that maybe __package__ value is wrong, but
apparently it is not. My guess now is that the "__path__" of the top level namespace wins 
and prefers the namespace directory, but that won't prevent it from importing compiled 
included stuff below it again.

The first failure is probably to not load the included namespace package.

Yours,
Kay
msg2476 (view) Author: kayhayen Date: 2018-08-10.10:48:38
Thanks for the reproducer:

python3 ../bin/nuitka --recurse-all --show-module minimal_imports_example.py 

Nuitka:INFO:Recurse to import 'foo.foo' from 'foo/foo.py'. (Requested to recurse to all non-standard library modules.)
Nuitka:INFO:Recurse to import 'foo.bar.bar' from 'foo/bar/bar.py'. (Requested to recurse to all non-standard library modules.)
Nuitka:INFO:Recurse to import 'bar' from 'foo/bar'. (Containing package of recursed module 'foo.bar.bar'.)
Nuitka:INFO:Recurse to import 'foo' from 'foo'. (Containing package of recursed module 'foo.foo'.)
Nuitka:INFO:Included compiled module '__main__'.
Nuitka:INFO:Included compiled module 'bar'.
Nuitka:INFO:Included compiled module 'foo'.
Nuitka:INFO:Included compiled module 'foo.bar.bar'.
Nuitka:INFO:Included compiled module 'foo.foo'.


So this looks good to me. Also expanding the test was doing it. But then executing with "foo" moves out of sight indeed gave this:

  File "... minimal_imports_example.py", line 2, in <module>
    from foo.bar.bar import Bar
ImportError: No module named 'foo.bar'

So, what I am guessing to be happening, is that the namespace package fails to properly integrate with Nuitka compiled module imports, 
or at least other namespace packages.

Not sure at this point, how hotfixable it is going to be. But I am considering a new release anyway, but I am definitely going to fix 
this next. In the mean time, you can workaround it by adding the "__init__.py" files.

Yours,
Kay
msg2474 (view) Author: morefigs Date: 2018-08-10.02:14:47
Cheers for the quick response.

The package_missing_init example is different in that it only has a single subfolder. Attached is an example with nested subfolders that shows the issue:

- running minimal_imports_example.py directly works fine
- compiling and running minimal_imports_example.exe works fine
- compiling and deleting/renaming the foo folder and then running minimal_imports_example.exe gives the ModuleNotFoundError

Adding __init__.py files to *all* of the subfolders fixes the issue for me.

Versions:
Windows 10 64-bit version 1803
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 02:47:15) [MSC v.1900 32 bit (Intel)] on win32
Nuitka 0.5.32.3
msg2471 (view) Author: kayhayen Date: 2018-08-09.11:44:07
As an aside, the warning about _frozen_importlib_external has been added to the 
white listing, every name space package will get that, but it's OK that way.
msg2470 (view) Author: kayhayen Date: 2018-08-09.11:43:05
Just tried it myself:

$ /c/Python37_32/python bin/nuitka --recurse-all tests/programs/package_missing_init/
PackageMissingInitMain.py  some_package/

$ /c/Python37_32/python bin/nuitka --recurse-all --run tests/programs/package_missing_init/PackageMissingInitMain.py
Nuitka:WARNING:C:\Users\kayha\repos\Py2C\tests\programs\package_missing_init\some_package:1: Cannot find 
'_frozen_importlib_external' in package 'some_package' as absolute import (tried _frozen_importlib_external).
This must be Python3.3 or higher, which no longer needs __init__.py to accept a package.
The parent path is _NamespacePath(['C:\\Users\\kayha\\repos\\Py2C\\tests\\programs\\package_missing_init\\some_package'])

$ rm -rf tests/programs/package_missing_init

$ ./PackageMissingInitMain.exe
This must be Python3.3 or higher, which no longer needs __init__.py to accept a package.
The parent path is _NamespacePath(['C:\\Users\\kayha\\repos\\Py2C\\tests\\programs\\package_missing_init\\some_package'])

So that appears to work for me. The message is from that submodule in that namespace package. 

The name space package has a __path__ that points to where it lives. And why would it not, that is correct. That is what 
namespace packages are for, to point to one or more locations to import from. If you do import in a way that is not visible to 
Nuitka, with __import__, only then it will load from there. You might want to checkout --include-module and --include-package in 
these cases. You probably got a warning along those lines that you ignored.

So I think this issue is not a bug.

Yours,
Kay

Yours,
Kay
msg2467 (view) Author: kayhayen Date: 2018-08-09.10:39:03
Hello,

do you have a minimal reproducer to prove this? Our put otherwise, how is it 
different from this:

find tests/programs/package_missing_init/
tests/programs/package_missing_init/
tests/programs/package_missing_init/PackageMissingInitMain.py
tests/programs/package_missing_init/some_package
tests/programs/package_missing_init/some_package/some_module.py

That seems to work in my tests. Are you saying they secretely load uncompiled 
code behind my back?

Yours,
Kay
msg2465 (view) Author: morefigs Date: 2018-08-09.08:01:58
Python 3.3+ no longer requires an __init__.py file within modules. However if these files aren't present Nuitka seems to ignore these modules when compiling. The executable will 
still run if the source modules are in place, but if they're removed then the executable fails with: "ModuleNotFoundError: No module named X".
History
Date User Action Args
2018-08-16 09:17:50kayhayensetstatus: testing -> resolved
messages: + msg2481
2018-08-13 08:59:16kayhayensetmessages: + msg2480
2018-08-13 02:35:12morefigssetmessages: + msg2479
2018-08-10 14:14:51kayhayensetstatus: chatting -> testing
messages: + msg2478
keyword: + wrong_execution
2018-08-10 13:55:27kayhayensetmessages: + msg2477
2018-08-10 10:48:38kayhayensetmessages: + msg2476
2018-08-10 10:33:00kayhayensetassignedto: kayhayen
nosy: + kayhayen
2018-08-10 10:32:31kayhayensettitle: Python modules without __init__.py file aren't included in build -> Nested namespace packages aren't included
2018-08-10 02:14:47morefigssetfiles: + minimal-imports-example.zip
messages: + msg2474
2018-08-09 11:44:07kayhayensetmessages: + msg2471
2018-08-09 11:43:05kayhayensetmessages: + msg2470
2018-08-09 10:39:03kayhayensetstatus: unread -> chatting
messages: + msg2467
2018-08-09 08:01:59morefigscreate