Issue243

Title MacOS: IOError DLL not found on OSX Anaconda Python
Priority bug Status resolved
Superseder Nosy List cjrh, kayhayen
Assigned To cjrh Keywords macos

Created on 2015-09-09.03:41:56 by cjrh, last changed by kayhayen.

Messages
msg1951 (view) Author: kayhayen Date: 2016-08-16.10:38:38
No feedback, I am assuming it is solved.
msg1578 (view) Author: kayhayen Date: 2015-09-26.10:40:07
Please retry with pre5, it should contain a fix.
msg1557 (view) Author: cjrh Date: 2015-09-19.02:25:10
Ok, I tried it out. I seem to get the same problem, so maybe I haven't correctly obtained the new code.  I did:

First, create new conda environment:
=====================================================

calebhattingh $ conda create -n n34test python=3.4 pip
Fetching package metadata: ....
Solving package specifications: .
Package plan for installation in environment /Users/calebhattingh/anaconda/envs/n34test:

The following NEW packages will be INSTALLED:

    openssl:    1.0.1k-1
    pip:        7.1.2-py34_0
    python:     3.4.3-0
    readline:   6.2-2
    setuptools: 18.1-py34_0
    sqlite:     3.8.4.1-1
    tk:         8.5.18-0
    wheel:      0.24.0-py34_0
    xz:         5.0.5-0
    zlib:       1.2.8-0

Proceed ([y]/n)? y

Linking packages ...
[      COMPLETE      ]|###################################################| 100%
#
# To activate this environment, use:
# $ source activate n34test
#
# To deactivate this environment, use:
# $ source deactivate

Active the new env
=====================================================

calebhattingh $ . activate n34test
discarding /Users/calebhattingh/anaconda/bin from PATH
prepending /Users/calebhattingh/anaconda/envs/n34test/bin to PATH

Install the development version of Nuitka from github
=====================================================

calebhattingh $ pip install git+git://github.com/kayhayen/Nuitka.git#egg=Nuitka
Collecting Nuitka from git+git://github.com/kayhayen/Nuitka.git#egg=Nuitka
  Cloning git://github.com/kayhayen/Nuitka.git to /private/var/folders/kn/7ynr8kzd321_2_fbkw150rtc0000gn/T/pip-build-
waedkdzw/Nuitka
Installing collected packages: Nuitka
  Running setup.py install for Nuitka
Successfully installed Nuitka-0.5.15rc4

Build my test.py again
======================

calebhattingh $ nuitka --recurse-all test.py

Try to run the file:
====================

calebhattingh $ ./test.exe
dyld: Library not loaded: libpython3.4m.dylib
  Referenced from: /Users/calebhattingh/Dropbox/Technical/codelibs/workspace/scratch/nuitka/mac/./test.exe
  Reason: image not found
Trace/BPT trap: 5

Examine the linking
===================

calebhattingh $ otool -L test.exe
test.exe:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
	libpython3.4m.dylib (compatibility version 3.4.0, current version 3.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

So it doesn't seem as if anything has changed. (Perhaps I got the wrong version of Nuitka?)

I tried to change the linking name myself, as experiment
========================================================

(I first copied the exe to "test.34.exe")

calebhattingh $ install_name_tool -change libpython3.4m.dylib /Users/calebhattingh/anaconda/envs/n34test/lib/libpython3.4m.dylib 
test.34.exe

Verify that the linking has changed:
====================================

calebhattingh $ otool -L test.34.exe
test.34.exe:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
	/Users/calebhattingh/anaconda/envs/n34test/lib/libpython3.4m.dylib (compatibility version 3.4.0, current version 3.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Try to run the executable now:
==============================

calebhattingh $ ./test.34.exe
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
Abort trap: 6

I am currently inside an activate conda env with all the right stuff installed.  I guess the executable is not aware of all of 
that.

Last try: what about --standalone
=================================

calebhattingh $ nuitka --standalone test.py
Traceback (most recent call last):
  File "/Users/calebhattingh/anaconda/envs/n34test/bin/nuitka", line 181, in <module>
    MainControl.main()
  File "/Users/calebhattingh/anaconda/envs/n34test/lib/python3.4/site-packages/nuitka/MainControl.py", line 758, in main
    standalone_entry_points = standalone_entry_points
  File "/Users/calebhattingh/anaconda/envs/n34test/lib/python3.4/site-packages/nuitka/freezer/Standalone.py", line 774, in 
copyUsedDLLs
    used_dlls = detectUsedDLLs(standalone_entry_points)
  File "/Users/calebhattingh/anaconda/envs/n34test/lib/python3.4/site-packages/nuitka/freezer/Standalone.py", line 685, in 
detectUsedDLLs
    assert Utils.isAbsolutePath(dll_filename), dll_filename
AssertionError: libpython3.4m.dylib



That also doesn't seem to work, and it looks like for similar reasons, something to do with locating the python lib.

Ideas?

What I would really, really love to do is run a one-liner on my python code, from inside a conda environment, and produce a 
completely standalone, I-can-copy-it-to-another-computer-and-it-just-works kind of setup.  I'm happy to help, but I don't have a 
lot of free time unfortunately :(
msg1548 (view) Author: cjrh Date: 2015-09-14.23:37:50
Sorry, not yet.
msg1546 (view) Author: kayhayen Date: 2015-09-14.20:01:56
Did you have a chance to check it out?

Thanks,
Kay
msg1531 (view) Author: kayhayen Date: 2015-09-12.09:40:20
I including the "otool" usage in the latest pre-release. Please try that out, 
and let me know if it works better.

For binaries to copy to other machines, use --standalone, then it will work. 
It produces a .dist folder with an .exe inside, which then is to be copied as 
a whole, static linking couldn't really work that completely, so it's not 
done, we just use and include the Python and system DLLs.

Yours,
Kay
msg1521 (view) Author: cjrh Date: 2015-09-09.05:41:38
Firstly, my Anaconda python is installed in my home folder, which is the recommended way, 
and secondly, my python 3.4 binary, and lib, is in a conda virtual path. It's like 
virtualenv, but Anaconda's version of that.  So I think you would have to look up the path 
to the lib at runtime (Is that what you meant?).  It doesn't seem too difficult: using the 
currently-active python (i.e. the one nuitka was installed into and has been run), 
sys.prefix will give the location of the python virtual environment, and then the dylib is 
inside the `lib/` folder off that.

So the lib location might be:

import sys
form os.path import join
libdir = join(sys.prefix, 'lib', 'libpython3.4m.dylib')

As a separate and possibly more direct way, you can also get the lib dir using distutils:

from distutils import sysconfig;
print sysconfig.get_config_var("LIBDIR")

I think the second way is probably better.  On my system that produces:

/Users/calebhattingh/anaconda/envs/py3/lib


Am I correct in assuming that the produced binary will only work on my machine, if I use 
nuitka in this way, i.e. to build against a python in a conda env, within my home folder?  
I haven't done much research here, but it is possible to statically link the python lib in?
msg1520 (view) Author: kayhayen Date: 2015-09-09.05:17:33
Oh, and maybe a chmod must be done before, not sure about that, the standalone 
code for MacOS does it. Maybe not necessary, but I wanted to mention it.
msg1519 (view) Author: kayhayen Date: 2015-09-09.05:15:45
So, what we are going to do for MacOS is to execute "install_name_tool" in the binary.

install_name_tool -change libpython3.4m.dylib /real/path/libpython3.4m.dylib test.exe 

that ought to work then, with no copying involved. Can you provide me with the path for 
the "python" binary and the python DLL as above on your system, so I can (blindly, as I 
have no MacOS) create code that will do this in the scons file.

That's going to be better than copying the DLL and makes me wonder, if the same cannot be 
achieved for Windows too, which would make it dramatically less ugly as well.

Yours,
Kay
msg1518 (view) Author: cjrh Date: 2015-09-09.04:22:53
I'm...unsure.  The possibilities keep changing all the time.  On the current 
"Install" page, the mac install shows screenshots. An option is there to install for 
all users but it's greyed out. There are discussions about a system wide install for 
mac in the groups, but I could not say what is doctrine.

http://docs.continuum.io/anaconda/install#mac-install
msg1517 (view) Author: kayhayen Date: 2015-09-09.04:17:45
Ok. So for MacOS we could use otool to set the proper path. Much like done
for standalone. Then no copy will be needed.

For Windows I vaguely recall the installer to make it a choice of.the user
to install system wide. Not the case on Mac?
msg1516 (view) Author: cjrh Date: 2015-09-09.04:12:10
I made a new conda env with Python 2.6, and that worked fine.  This suggests 
something is different with Python 3, rather than a conda env.

For my Python 3.4 conda env, I verified that libpython3.4m.dylib is definitely 
present in envs/py3/lib/

So that's weird.
msg1515 (view) Author: cjrh Date: 2015-09-09.04:00:49
Ok, there are complications.  There is no problem for Python 2.7, but there is a problem for Python 3.4.

*It might be because my Python 3.4 is in a Conda env*. My Python 2.7 is the root env for my anaconda 
installation.

For 3.4:
########

Compilation succeeds, reports no errors.

Running test.exe produces:

calebhattingh $ ./test.exe
dyld: Library not loaded: libpython3.4m.dylib
  Referenced from: /Users/calebhattingh/Dropbox/Technical/codelibs/workspace/scratch/nuitka/mac/./test.exe
  Reason: image not found
Trace/BPT trap: 5

otool produces:

calebhattingh $ otool -L test.34.exe
test.34.exe:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
        libpython3.4m.dylib (compatibility version 3.4.0, current version 3.4.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
msg1514 (view) Author: cjrh Date: 2015-09-09.03:54:41
It seems it does not need the copy, correct.  Here's otool:

calebhattingh $ otool -L test.exe
test.exe:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
        libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

For reference, test.py is:

import os
print(os.listdir('.'))
msg1513 (view) Author: kayhayen Date: 2015-09-09.03:52:02
So, that test.exe does not need the copy? If that's the case, then there is just an 
os.name checking missing. If that's not the case, some ".so" needs to be copied. But the 
rpath equivalent might make this unnecessary. It would be sweet if you pasted the output 
of otool for test.exe. 

Just let me know, and I can put this into the next hotfix.

Yours,
Kay
msg1512 (view) Author: cjrh Date: 2015-09-09.03:44:24
I should add that by manually setting `win_copy_dll` to "false", the compilation works 
and I get a working "test.exe" out.
msg1511 (view) Author: cjrh Date: 2015-09-09.03:41:56
Please see:

https://github.com/kayhayen/Nuitka/search?utf8=%E2%9C%93&q=win_copy_dll&type=Code

I am running Anaconda Python, so `isUninstalledPython()` returns true, however I'm 
not running Windows (this bug is on OSX), so there is no DLL to copy.

The error that results looks like this below. The same occurs on Python 3.4 
(Anaconda). 

calebhattingh $ nuitka test.py
IOError: [Errno 2] No such file or directory: 
'/Users/calebhattingh/anaconda/python27.dll':
  File "/Users/calebhattingh/anaconda/lib/python2.7/site-
packages/nuitka/build/SingleExe.scons", line 1414:
    os.path.dirname(result_basepath) or "."
  File "/Users/calebhattingh/anaconda/lib/python2.7/shutil.py", line 119:
    copyfile(src, dst)
  File "/Users/calebhattingh/anaconda/lib/python2.7/shutil.py", line 82:
    with open(src, 'rb') as fsrc:
History
Date User Action Args
2016-08-16 10:38:38kayhayensetstatus: testing -> resolved
messages: + msg1951
2015-09-26 10:40:07kayhayensetmessages: + msg1578
2015-09-19 02:25:10cjrhsetmessages: + msg1557
2015-09-14 23:37:50cjrhsetmessages: + msg1548
2015-09-14 20:01:56kayhayensetmessages: + msg1546
2015-09-13 10:11:32kayhayensetassignedto: kayhayen -> cjrh
2015-09-12 09:40:26kayhayensetstatus: chatting -> testing
2015-09-12 09:40:20kayhayensetmessages: + msg1531
2015-09-09 05:41:38cjrhsetmessages: + msg1521
2015-09-09 05:17:33kayhayensetmessages: + msg1520
2015-09-09 05:15:45kayhayensetmessages: + msg1519
2015-09-09 04:22:53cjrhsetmessages: + msg1518
2015-09-09 04:17:45kayhayensetmessages: + msg1517
2015-09-09 04:12:10cjrhsetmessages: + msg1516
2015-09-09 04:00:49cjrhsetmessages: + msg1515
2015-09-09 03:54:41cjrhsetmessages: + msg1514
2015-09-09 03:52:20kayhayensetkeyword: + macos
title: IOError DLL not found on OSX Anaconda Python -> MacOS: IOError DLL not found on OSX Anaconda Python
2015-09-09 03:52:02kayhayensetassignedto: kayhayen
messages: + msg1513
nosy: + kayhayen
2015-09-09 03:44:24cjrhsetstatus: unread -> chatting
messages: + msg1512
2015-09-09 03:41:56cjrhcreate