Title Nuitka breaks PyQt5 threads on python3 (help wanted)
Priority bug Status chatting
Superseder Nosy List hawk
Assigned To Keywords help_wanted, wrong_execution

Created on 2015-04-09.14:57:48 by hawk, last changed by kayhayen.

File name Uploaded Type Edit Remove hawk, 2015-04-09.14:57:48 application/x-python
msg1817 (view) Author: kayhayen Date: 2016-02-12.12:19:08
From PyQt5 source in my Debian testing right now: qpy/QtCore/qpycore_pyqtslot.cpp

This is the most likely candidate for the problematic code:

// Create the slot for a callable.
PyQtSlot::PyQtSlot(PyObject *callable, const Chimera::Signature *slot_signature)
    : mfunc(0), mself_wr(0), other(0), signature(slot_signature)
    if (PyMethod_Check(callable))
        // Save the component parts.
        mfunc = PyMethod_GET_FUNCTION(callable);
        mself = PyMethod_GET_SELF(callable);
        mclass = PyMethod_GET_CLASS(callable);

        // Try and create a weak reference to the instance object.
        mself_wr = PyWeakref_NewRef(mself, 0);
        // Give the slot an extra reference to keep it alive.
        other = callable;

My suspect, is that by just using the compiled method as that slot callable, any checks for the 
instance "mself" if it has a thread attached, will not be done, resulting in the threading problem.

Similar things have been done before, but I couldn't locate my patch anymore. It seems that PyQT got 
much better at avoiding these types of hard type checks, as there is barely any use of 
PyFunction_Check and that's guarded by PyMethod_Check, which also only has very few uses.

I might be able to produce very simple patches to improve PyQt5 here. In the past upstream has 
accepted similar patches, so stay tuned.

msg1815 (view) Author: kayhayen Date: 2016-02-12.11:16:35
No idea, what made me think that in the original report, exceptions play a role, 
it's just a mere PyQt method, used as a slot that checks the thread of the 
object and finds it's different from the currently running one.

So this is purely about Qt threading in PyQt not working for compiled code. I 
did not recurse into Qt at all, and I see the issue. So it must be suspected, 
that "moveToThread()" or "@QtCore.pyqtSlot" do something different if it's not a  

I am going to check their sources for any indications of that. Might need more 
PyQt patches.

msg1312 (view) Author: marc.jofre Date: 2015-06-01.20:41:59
Hi all,

My aim is to provide some insights related to this issue: PyQt5 threads errors

Could nuitka recover from unexpected python thread errors???

While testing regular python execution (python console, without nuitka) I came 
across with an execution error with multiprocessing (error dump shown below). 
Therefore I have upgraded multiprocessing script (pip install multiprocessing --

I believe it has relation with this issue because it is related to errors in 
Threads which python already struggles with them. Although, python recovers from 
this errors and it is quite transparent to the user (I have only experimented 
this error in a new computer with a slightly diferent version of winpython).

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\WinPython-64bit-\python-2.7.6.amd64\lib\multiprocessing\forkin", line 380, in main
  File "C:\WinPython-64bit-\python-2.7.6.amd64\lib\multiprocessing\forkin", line 495, in prepare
    '__parents_main__', file, path_name, etc
  File "c:\SixSenso\productgeneraldistributables\backend\Backend_GUI\mysite\mysi
te\", line 48, in <module>
    from twisted.python import log
  File "C:\WinPython-64bit-\python-2.7.6.amd64\lib\site-packages\twisted\
python\", line 22, in <module>
    from twisted.python import util
  File "C:\WinPython-64bit-\python-2.7.6.amd64\lib\site-packages\twisted\
python\", line 9, in <module>
    import pwd, grp
msg1251 (view) Author: kayhayen Date: 2015-05-01.10:43:44
I am asking on the mailing list for help. This will need some investigation.

The thing is that with Python3 potentially, or generally it is possible to raise an exception to specific threads. Nuitka sets it 
async, and then just any thread will take it. I think that was Python2 compatible, but maybe it never was.

The code is added in loops. Just do a grep for "CONSIDER_THREADING". It will not be done otherwise:

    // Decrease ticker
    if ( --_Py_Ticker < 0 )
        _Py_Ticker = _Py_CheckInterval;

        int res = Py_MakePendingCalls();

        if (unlikely( res < 0 && ERROR_OCCURRED() ))
            return false;

        PyThreadState *tstate = PyThreadState_GET();
        assert( tstate );

        if ( PyEval_ThreadsInitialized() )
            // Release and acquire the GIL, it's very inefficient, because we
            // don't even know if it makes sense to do it. A controlling thread
            // should be used to determine if it's needed at all.
            PyEval_AcquireThread( tstate );

        if (unlikely( tstate->async_exc != NULL ))
            PyObject *async_exc = INCREASE_REFCOUNT( tstate->async_exc );
            tstate->async_exc = NULL;

            RESTORE_ERROR_OCCURRED( async_exc, NULL, NULL );

            return false;

    return true;

Maybe somebody can find out, if that async_exc contains any information to use and decide tha
an exception is not for that thread. But since it's in the tstate, read thread state, something
went wrong, when it was raised, which Nuitka likely doesn't do.

msg1148 (view) Author: hawk Date: 2015-04-11.21:56:54
Qt has facilities to implement "remote" calls across threads using the signals/slots 

For reference you can have a look at the example here:

and at the QueuedConnection description here:

If you fire a signal "QueuedConnected" to a slot belonging to a QObject living in a 
remote thread's main-loop (see, 
then you are guaranteed that the slot is executed from that main-loop (i.e.: in that 
msg1147 (view) Author: kayhayen Date: 2015-04-11.19:10:41
I wonder, does PyQt give such a thing as a promise to be in the same thread.

That said, it's entirely likely that the async exception mechanism changed in 
Python3 so that it's possible to raise an exception to a specific thread, I will 
need to check that out then.
msg1139 (view) Author: hawk Date: 2015-04-09.14:57:48
$ python3 --version
Python 3.4.0

$ nuitka --version

$ sip -V

$ python3 -c 'from PyQt5 import QtCore; print(QtCore.PYQT_VERSION_STR)'

$python3 /usr/bin/nuitka --recurse-not-to PyQt5 && ./test.exe

When running the attached test file, before and after compilation with nuitka,
the behavior changes.

In Qt5 (and PyQt5), when you fire a queued signal connected to a QObject's slot
living on another thread's main loop, the slot is called in the context of the
that thread. But when you compile it with niutka the signal does invoke the slot
from the same thread which emits it.

The testSlot in the example is running in the wrong thread when compiled.
Date User Action Args
2016-02-12 12:19:08kayhayensetmessages: + msg1817
2016-02-12 11:16:35kayhayensetmessages: + msg1815
2015-06-01 20:41:59marc.jofresetmessages: + msg1312
2015-05-01 10:43:44kayhayensetmessages: + msg1251
2015-05-01 10:38:31kayhayensetkeyword: + wrong_execution, help_wanted
title: Nuitka breaks PyQt5 threads on python3 -> Nuitka breaks PyQt5 threads on python3 (help wanted)
2015-04-16 20:04:14kayhayensetpriority: critical -> bug
2015-04-11 21:56:54hawksetmessages: + msg1148
2015-04-11 19:10:41kayhayensetstatus: unread -> chatting
messages: + msg1147
2015-04-09 14:57:48hawkcreate