Title Scons config misses python binary under Anaconda
Priority bug Status in-progress
Superseder Nosy List kayhayen, kyzyl
Assigned To kayhayen Keywords windows

Created on 2016-02-13.06:41:33 by kyzyl, last changed by kayhayen.

msg1829 (view) Author: kayhayen Date: 2016-02-15.20:15:22
I have managed to find the running DLL with ctypes. It's needed for MinGW64 and to copy it along 
for "uninstalled" Python versions still. The first use won't go away unfortunately, the second I 
am currently working on.

Also the "python.exe" is now passed from the running Python instance, so it will be more likely 

Also, switching to debug python or not with CPython3.5 works.

This has removed a bunch of code from "SingleExe.scons" that was only guessing.

The "pythonXY.lib" is searched in "libs" and "PCbuild" both, and the respective path that has it, 
is used.

I just need to figure out, how to make the accelerated binary add the DLL path to the DLL 
directories, while working with unicode paths and all kinds of things, then it ought to be really 

I am hoping for a pre-release with this, factory is undergoing more and other changes and 
relatively broken otherwise, but might contain something usable soonish.

msg1827 (view) Author: kayhayen Date: 2016-02-15.10:08:41
I have installed latest Anaconda 2.5.0, 64 bit, the Python2.7 variant. And this command yields nothing:

find ~/Anaconda2/ -iname PCBuild

Can it be, that this only comes into existence, once you conda install something that requires a C 
extension build? What are we missing there, I am sure this does not affect all users. Therefore it 
probably makes sense to put changes into a pre-release rather than a hotfix. Can you check the file 
date of your PCBuild directory, to see if it's different from the rest of Anaconda install to confirm 
my theory?

Anyway, I will follow the plan and remove the detection based on "PCBuild" presence, and instead scan 
for paths. I was wondering, if it were not best for the Python.exe that ran Nuitka to pass the paths to 
Python.DLL and Python.EXE as paths, instead of using python_prefix to guess. This is the best idea, as 
the running binary will know best what it did.

And I would love to finally achieve compilation of the "Python.DLL" path into the accelerated binary, 
as even CPython's Python3.5 installs now cannot find Python.DLL anymore by themselves. That would 
remove one job for "uninstalled" Python and in my mind make them first class citizens with Nuitka for 
acceleration mode.

BTW: Also it might be interesting to revisit, if MinGW64 cannot work with existing ".LIB" files 
nowadays, in which case, combined with the above statement, uses of the exact ".DLL" path might be 
all removed from Scons. For the above, it would be in the created source code only. No need for Scons 
to use it anymore.

First step obviously will be to create Buildbot jobs that check that Anaconda and WinPython both work 
with basic tests. Then to make the DLL path compiled into the binary, making the copying of the DLL not 
necessary. For that, a clever way to find the used Python.DLL path in the list of loaded DLLs at Nuitka 
run time instead of scons time would be needed. I suspect, some ctypes usage over loaded modules with 
Win32 api will do that easily.

Then I can try to eliminate the "gendefs.exe" usage for MinGW64 64 bit, and if that's not possible, 
pass the DLL path to scons as well, so it always uses the correct one. Maybe it's no longer needed or 
not for all Python versions. I kind of hate to make that "gendefs.exe" call for each compilation.

When done, this will be an all around improvement of Nuitka for Windows. I am treating this as highest 
priority, as it's kind of a legacy to get rid off these issues once and for all in a portable way. The 
people want to choose the Python version they want to.

msg1825 (view) Author: kayhayen Date: 2016-02-13.09:23:55

thanks for the detailed report. Once back I will try and look into this. This 
might be a regression of Nuitka or a change of AnaConda recently to have this 
directory, I know this was working before.

Generally it might be better to not have that mode, but instead use a candidate 
list of paths to check for existance, before going ahead, with the "PCBuild" 
ones as a last resort anyway.

I currently have no access to a Windows machine, but I can sort of promise some 
kind of hotfix that won't expose such issue anymore for next week.

msg1824 (view) Author: kyzyl Date: 2016-02-13.06:41:33
Apologies if I'm missing something very simple here, but on my system (Win7x64, 
Python2.7 64bit, Anaconda) Nuitka refuses to see my python binary, failing with 
an assertion error on even a simple "Hello World" program.

AssertionError: C:\<snip>\Anaconda\PCBuild\python.exe:
  File "C:\<snip>\Anaconda\lib\site-packages\nuitka\build\SingleExe.scons", line 
    assert os.path.exists(python_exe), python_exe

I've traced the behavior to the SingleExe.scons file. In this config, scons 
attempts to locate the python binary, and seems to be making allowances for 
python installations built from source via the win_from_source_mode flag:

win_from_source_mode = win_target and \
                       os.path.exists(os.path.join(python_prefix, "PCBuild"))

In the case of my Anaconda distribution there exists a PCBuild folder, presumably 
because Continuum builds Anaconda from source, however it doesn't contain a 
python binary. Later on in the scons config the win_from_source_mode (which 
evaluates to True under Anaconda) is checked:

if win_from_source_mode:
     win_lib_path = os.path.join(python_prefix, "PCBuild")

if win_from_source_mode:
     # Also consider a Python built from source.
     python_exe = os.path.join(
         "python_d.exe" if python_debug else "python.exe"

leading to the selection of a python binary which doesn't exist. Obscuring the 
PCBuild folder from Nuitka's view and then compiling results in the correct 
selection, and Nuitka runs properly, producing a working binary.
Date User Action Args
2016-02-15 20:15:31kayhayensetassignedto: kayhayen
nosy: + kayhayen
2016-02-15 20:15:22kayhayensetmessages: + msg1829
2016-02-15 10:08:41kayhayensetstatus: chatting -> in-progress
messages: + msg1827
keyword: + windows
2016-02-13 09:23:55kayhayensetstatus: unread -> chatting
messages: + msg1825
2016-02-13 06:41:33kyzylcreate