Issue272

Title Several troubles with OSX
Priority bug Status resolved
Superseder Nosy List kayhayen, rboulanger
Assigned To kayhayen Keywords

Created on 2015-12-06.20:33:59 by rboulanger, last changed by kayhayen.

Files
File name Uploaded Type Edit Remove
__constants.cpp rboulanger, 2015-12-07.11:02:58 application/octet-stream
__constants.cpp_linux.cpp rboulanger, 2015-12-07.11:03:59 application/octet-stream
__frozen.cpp rboulanger, 2015-12-07.11:02:26 application/octet-stream
__frozen.cpp_linux.cpp rboulanger, 2015-12-07.11:03:31 application/octet-stream
out3 kayhayen, 2015-12-29.12:41:38 application/octet-stream
Messages
msg2324 (view) Author: kayhayen Date: 2018-03-14.13:37:28
Something like this got implemented and appears to work.
msg1747 (view) Author: kayhayen Date: 2015-12-29.13:53:00
As for loader_path, check this out:

http://stackoverflow.com/questions/16826922/what-path-does-loader-path-resolve-to

Seems as if it's just takes a os.path.join of what's after the @loader_path with the dirname of 
the library Nuitka is looking at. If you find the time, please hack that up and send it to me. 
Otherwise, eventually I will try/hack that without a system to test... but seems trivial.

Yours,
Kay
msg1746 (view) Author: kayhayen Date: 2015-12-29.12:41:38
This, this is the differences between Linux and MacOS, and it comes to freezing, and it shows that there is tests being 
installed on your MacOS python and other stuff:

While this is probably good stuff:

< ctypes.macholib
< ctypes.macholib.dyld
< ctypes.macholib.dylib
< ctypes.macholib.framework

There is also distutils.tests with a lot of modules, where some are big, but all are not going to be used.

< distutils.tests
< distutils.tests.support
< distutils.tests.test_archive_util
...

One big offender might be this:

< ensurepip
< ensurepip._uninstall

Not sure, but ensurepip could be relatively large, if it bundles stuff. Surely it needs not be bundled though.

Then this:

< sqlite3.test
< sqlite3.test.dbapi
< sqlite3.test.dump
< sqlite3.test.factory
< sqlite3.test.hooks
< sqlite3.test.regression
< sqlite3.test.transactions
< sqlite3.test.types
< sqlite3.test.userfunctions

That is probably not small either, also not needed surely. 

This:

< tkinter
< tkinter._fix
< tkinter.colorchooser
< tkinter.commondialog
< tkinter.constants
< tkinter.dialog
< tkinter.dnd
< tkinter.filedialog
< tkinter.font
< tkinter.messagebox
< tkinter.scrolledtext
< tkinter.simpledialog
< tkinter.tix
< tkinter.ttk

Could probably well be used, and will have to be retained.

Then this:

< unittest.test
< unittest.test._test_warnings
< unittest.test.dummy
< unittest.test.support
< unittest.test.test_assertions

Absolutely not needed.

Arguably, having these installed is almost worth bug reports to distribution of Python you are using. You end up with 8MB+ of 
bytecode, which then is channeled through C code, which apparently doesn't compile fast. For Linux, we have the ability to 
directly turn blobs into object files without C. It would be sweet to do that for MacOS too, but I don't know how to do it. 
Check out SingleExe.scons to see where to plug this, if you manage to find out.

For now, I am merely going to blacklist these module names from automatic inclusion, then these are going to be faster, expect 
the next prerelease to contain those.

If you have any news on @rpath, it would be good. This seems to be a widespread issue.

Yours,
Kay
msg1729 (view) Author: rboulanger Date: 2015-12-07.11:07:01
>>For the absolut path issue, Qt is compiled strangely it seems.

The first attempt with anaconda python, qt was taken from anacondas repositories,
in my second attemt on a clean osx install I compiled it self (like I did also under 
linux) but in the normal way. under all linux versions I tried, it worked like a charm.
but I will search for the @rpath thing now
msg1728 (view) Author: rboulanger Date: 2015-12-07.11:03:59
and the constants.cpp from sucessful linux build
msg1727 (view) Author: rboulanger Date: 2015-12-07.11:03:31
The frozen from the successful linux build
msg1726 (view) Author: rboulanger Date: 2015-12-07.11:02:58
the __constants.cpp from OSX build
msg1725 (view) Author: rboulanger Date: 2015-12-07.11:02:26
here the frozen.cpp from the OSX build
msg1724 (view) Author: kayhayen Date: 2015-12-07.00:40:57
For the constants issue, would it be possible for you to make a diff of the 
__constants.cpp maybe, that's maybe going to be very revealing.

Also the bytecode freezing might pick up way too many modules, if it is confused about 
what is standard library path and what not. Were it to pick up all the bytecode you 
got, that would explain the size. I can decide this if you attached the __frozen.cpp 
from the MacOS build directory.

Thanks,
Kay
msg1723 (view) Author: kayhayen Date: 2015-12-07.00:35:23
For the absolut path issue, Qt is compiled strangely it seems.

Please hack _detectBinaryPathDLLsMacOS in either ways:

a) Collect the problematic otool output and paste it here.
b) Optionally read up on what @rpath/ means, and make it resolve to
   the correct, absolute path.

Yours,
Kay
msg1722 (view) Author: rboulanger Date: 2015-12-06.21:01:13
AddOn:
Anyway, I tried to start the executable on the machine where it was compiled on. It 
starts up, but after a minute or so (I even didn't touch it) it dies because an 
unhandled exception.

Copied it over to another Mac, tried to start it, it says:
dyld: Library not loaded: /Library/Frameworks/Python.framework/Versions/3.4/Python
  Referenced from: /Volumes/Macintosh 
HD/Development/python3/rtWriterMac.dist/./rtWriter
  Reason: image not found
Trace/BPT trap: 5

maybe this helps
msg1721 (view) Author: rboulanger Date: 2015-12-06.20:33:59
So, there are several issues, which may have one cause.
First, I can compile and execute my program without any troubles under 

windows10, python 3.5, PyQt 5.5.1
linux kernel 3.2, python 3.4.3, PyQt 5.5.1
linux kernel 4.2, python 3.4.3, PyQt 5.5.1

the command used to compile was in all platforms:

nuitka --standalone --windows-disable-console --jobs=8 --show-progress --show-scons --verbose --plugin-enable=qt-plugins rtWriter.py


there is no other dependency, just pure python, pure pyqt5.5, everything works fine with nuitka 0.5.16.1

under Mac OSX I tried several paths:

Yosemite, anaconda python 3.4.3, Qt pyqt5.5.1, nuitka 0.5.16.1:

first it takes on a 8-core MacPro about 6 (yes, six!) hours for this single scons statement 

scons: building `/Users/robert/Development/Writerdist/rtWriter.build/__constants_data.o' because it doesn't exist
gcc -o /Users/robert/Development/Writerdist/rtWriter.build/__constants_data.o -c -w -fvisibility=hidden -fvisibility-inlines-hidden -fvisibility=hidden -O3 -pipe 
-D_XOPEN_SOURCE -D_NUITKA_FULL_COMPAT -D_NUITKA_STANDALONE -D__NUITKA_NO_ASSERT__ -D_NUITKA_FROZEN=664 -D_NUITKA_MODULE_COUNT=74 -D_NUITKA_EXE -
I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -I/Users/robert/Development/Writerdist/rtWriter.build -
I/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/nuitka/build/include 
/Users/robert/Development/Writerdist/rtWriter.build/__constants_data.c

__constants_data.c contains a single 47 MB Array which looks like this:

const unsigned char constant_bin[] =
{
    0x6e, 0x61, 0x6d, 0x65, 0x20, 0x27, 0x73, 0x79, 0x73, 0x27, 0x20, 0x69, 0x73, 0x20, 0x6e, 0x6f,
    0x74, 0x20, 0x64, 0x65, 0x66, 0x69, 0x6e, 0x65, 0x64, 0x6e, 0x61, 0x6d, 0x65, 0x20, 0x27, 0x51,
....
}

after the 6 hours it alle ends up as described by cjrh in http://bugs.nuitka.net/issue243 

I retried with nuitka 0.5.14, 0.5.16.1 as well as with the 0.5.17 development version

so I stepped over to a clean 

El Capitan, plain python 3.4.3 from python.org, pyqt 5.5.1 and nuitka 0.5.16.1

again 6 hours busy with the scons shown above. 
when it ends this time nuitka shows:

Nuitka:INFO:Copying all Qt plug-ins to '../Writerdist/rtWriter.dist/PyQt5/qt-plugins'.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/bin/nuitka", line 193, in <module>
    MainControl.main()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/nuitka/MainControl.py", line 746, in main
    standalone_entry_points = standalone_entry_points
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/nuitka/freezer/Standalone.py", line 798, in copyUsedDLLs
    used_dlls = detectUsedDLLs(standalone_entry_points)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/nuitka/freezer/Standalone.py", line 709, in detectUsedDLLs
    assert Utils.isAbsolutePath(dll_filename), dll_filename
AssertionError: @rpath/QtCore.framework/Versions/5/QtCore


So the question is: what causes this final traceback and what is this 47 MB __constants_data.c which has just 88k under linux and looks there like this :
#include "nuitka/prelude.hpp"

// Sentinel PyObject to be used for all our call iterator endings. It will
// become a PyCObject pointing to NULL. It's address is unique, and that's
// enough for us to use it as sentinel value.
PyObject *_sentinel_value = NULL;

PyObject *const_int_0;
PyObject *const_float_1_0;
PyObject *const_int_neg_1;
PyObject *const_int_pos_1;
PyObject *const_int_pos_2;
PyObject *const_int_pos_3;
PyObject *const_int_pos_4;
PyObject *const_int_pos_5;
PyObject *const_int_pos_6;
PyObject *const_int_pos_7;
....
History
Date User Action Args
2018-03-27 10:36:53kayhayensetstatus: chatting -> resolved
2018-03-14 13:37:28kayhayensetmessages: + msg2324
2015-12-29 13:53:00kayhayensetassignedto: kayhayen
messages: + msg1747
2015-12-29 12:41:38kayhayensetfiles: + out3
nosy: + kayhayen
messages: + msg1746
2015-12-07 11:07:01rboulangersetmessages: + msg1729
2015-12-07 11:03:59rboulangersetfiles: + __constants.cpp_linux.cpp
messages: + msg1728
2015-12-07 11:03:31rboulangersetfiles: + __frozen.cpp_linux.cpp
messages: + msg1727
2015-12-07 11:02:58rboulangersetfiles: + __constants.cpp
messages: + msg1726
2015-12-07 11:02:26rboulangersetfiles: + __frozen.cpp
messages: + msg1725
2015-12-07 00:40:57kayhayensetmessages: + msg1724
2015-12-07 00:35:23kayhayensetmessages: + msg1723
2015-12-06 21:01:13rboulangersetstatus: unread -> chatting
messages: + msg1722
2015-12-06 20:33:59rboulangercreate