Title Nuitka is not working on musl libc
Priority bug Status deferred
Superseder Nosy List frol, kayhayen
Assigned To kayhayen Keywords

Created on 2016-08-16.18:40:21 by frol, last changed by ab1.

File name Uploaded Type Edit Remove
build_log.txt ab1, 2017-11-17.21:06:06 text/plain
msg2291 (view) Author: ab1 Date: 2017-11-17.21:06:06
I am attempting to build a --standalone package (python2.7, latest stable Nuitka) on Alpine Linux, so this might be related to musl 
compatibility issue, although the error messages I am getting are different.

I have installed pre-build glibc packages and confirmed the libraries are working by running SplunkUniversalForwarder, which is linked against 

When attempting to build my executable, the output I get is along the lines of:

# nuitka --recurse-all --improved --enhanced --remove-output\
    --output-dir=$(uname -m) --plugin-enable=pylint-warnings\

<...snip long list of warnings>
[Build]    Problem with statement at /usr/lib/python2.7/site-packages/eventlet/
[Build]    Problem with statement at /usr/lib/python2.7/site-packages/eventlet/
[Build]    Problem with statement at /usr/lib/python2.7/site-packages/eventlet/
[Build]    Problem with statement at /usr/lib/python2.7/site-packages/eventlet/
[Build]    Problem with statement at /usr/lib/python2.7/site-packages/eventlet/
[Build]    Problem with statement at /usr/lib/python2.7/site-packages/eventlet/
[Build]    Nuitka:INFO:Interrupted while working on '<Node 'COMPILED_PYTHON_MODULE' with {'package': 'eventlet', 'name': 'patcher', 'filename': 
[Build]    Traceback (most recent call last):
[Build]      File "/usr/bin/nuitka", line 211, in <module>
[Build]        MainControl.main()
<...snip long stack trace>
[Build]      File "/usr/lib/python2.7/site-packages/nuitka/nodes/", line 164, in computeStatement
[Build]        assert False, statement.asXmlText()
[Build]    AssertionError: <node is_temp="False" kind="StatementAssignmentVariable" line="376" 
owner="eventlet$patcher$$$function_13__fix_py3_rlock" variable_name="ref_vars">
[Build]      <role name="source">
[Build]        <node kind="ExpressionBuiltinVars" line="376">
[Build]          <role name="source">
[Build]            <node kind="ExpressionVariableRef" line="376" owner="eventlet$patcher$$$function_13__fix_py3_rlock" variable_name="ref" />
[Build]          </role>
[Build]        </node>
[Build]      </role>
[Build]    </node>

I've attached the full build log.

Any ideas?
msg2231 (view) Author: kayhayen Date: 2017-10-03.11:05:16
The fibers are going away. I have something going on that I dubbed "goto 
generators", which essentially will remove all that fiber and therefore context 
usage, and lower the requirements on libc in this regard.

Once this "goto generator" thing is done, this issue will then be automatically 
be resolved too. We can then clean up SingleExe.scons a lot.
msg2226 (view) Author: OpenWrtUser Date: 2017-09-19.05:15:47
any plans to move to pthreads?
msg2062 (view) Author: kayhayen Date: 2016-11-14.21:18:49
The order of checks in SingleExe.scons is wrong. The check for alpine is after the one for 
x86_64 linux. That is really bad. The reason I did that was because I didn't want a file 
system check at all for all systems. I need to. I would really love to have a check that 
is using sys.platform, uname, something from there to have better confidence. Of course 
that's non-sense, but file system based OS checks just feel entirely wrong to me.

msg2009 (view) Author: frol Date: 2016-10-29.18:16:58
Yay! `nuitka --module` succeeds to build a usable module on Alpine Linux!

However, it still cannot create a standalone executable (I have tried Nuitka 
0.23.1, 0.24rc2, and the develop branch) due to the following errors:

$ echo 'print("Hello")' >
$ nuitka In function `_prepareFiber':
fibers_x64.c:(.text+0x44): undefined reference to `getcontext'
fibers_x64.c:(.text+0xa3): undefined reference to `makecontext'
collect2: error: ld returned 1 exit status
scons: *** [start.exe] Error 1
msg2008 (view) Author: kayhayen Date: 2016-10-29.16:51:54
It's merge, and it should work now. Let me know if it doesn't.
msg1975 (view) Author: kayhayen Date: 2016-09-22.10:52:01
I didn't see any relevant diff to the versions already in Nuitka. In fact, the 
only diff seemed to be a fix for Win32 that is only in Nuitka. But I am going to 
merge the proposed patch, just as a different "elif", so it's not confusing 
later on.

It would be sweet to checkout if "libcoro" code would be better than generic 
implementation anyway. Then this could become the new default code and generic 
could be removed.

I am pushing this to factory branch in official git repo now. To be part of the 
next pre-release.
msg1974 (view) Author: nsiddiqui Date: 2016-09-22.03:13:34
I tried the following and it seems to work with musl (alpine)

1. Get coro.h and coro.c from
fibers/blob/musl/src/libcoro/ and copy to Nuitka-

2. Create a link to the coro.h file
ln -s .../Nuitka-0.5.22/nuitka/build/static_src/libcoro_ucontext_src/coro.h 

3. patch SingleExe.scons (this was specifically for alpine):
--- SingleExe.scons
@@ -1252,12 +1252,12 @@
     if win_target:
-    elif "openbsd" in sys.platform:
+    elif os.path.isfile('/etc/alpine-release') or "openbsd" in 
-            CPPDEFINES = ["CORO_SJLJ"]
+            CPPDEFINES = ["CORO_SJLJ", "__OpenBSD__"]
     elif target_arch == "x86_64" and "linux" in sys.platform:

hope this helps.
msg1954 (view) Author: kayhayen Date: 2016-08-17.06:43:58
Somebody will have to provide a fiber implementation that doesn't use context, 
there is already libcoro used in nuitka for OpenBSD which has the same issue.

Checkout in SingleExe.scons this:

    elif "openbsd" in sys.platform:

            CPPDEFINES = ["CORO_SJLJ"]

Something similar might do for musi too. I don't know. Please try it out and let 
me know. Maybe libcoro needs an update for more supported targets, but I think 
it ought to work as is.

msg1953 (view) Author: frol Date: 2016-08-16.18:40:21
Legacy functions operating on ucontext_t (getcontext, setcontext, makecontext, 
swapcontext) are not implemented in musl libc as they are no longer part of 
POSIX. Thus, Nuitka is not usable with musl libc (even though it manages to get 
.so library compiled and linked):

ImportError: Error relocating /mnt/build/ getcontext: symbol not 
Date User Action Args
2017-11-17 21:06:08ab1setfiles: + build_log.txt
messages: + msg2291
2017-10-03 11:05:16kayhayensetstatus: chatting -> deferred
messages: + msg2231
2017-09-19 05:15:47OpenWrtUsersetmessages: + msg2226
2016-11-14 21:18:49kayhayensetmessages: + msg2062
2016-10-29 18:16:58frolsetstatus: resolved -> chatting
messages: + msg2009
2016-10-29 16:51:54kayhayensetstatus: chatting -> resolved
messages: + msg2008
2016-09-22 10:52:01kayhayensetpriority: wish -> bug
messages: + msg1975
2016-09-22 10:45:57kayhayensetkeyword: - unsupported
2016-09-22 03:13:34nsiddiquisetmessages: + msg1974
2016-08-17 06:43:59kayhayensetstatus: unread -> chatting
keyword: + unsupported
nosy: + kayhayen
messages: + msg1954
priority: bug -> wish
assignedto: kayhayen
2016-08-16 18:40:21frolcreate