As discussed last week in TWN #1 this is a new series that I am using to highlight things that are going on, newly found issues, hotfixes, all the things Nuitka.
I made the first release with official 3.7 support, huge milestone in terms of catching up. Generic classes posed a few puzzles, and need more refinements for error handling, but good code works now.
The class creation got a bit more complex, yet again, which will make it even hard to know the exact base classes to be used. But eventually we will manage to overcome this and statically optimize that.
Building the MSI files for Nuitka ran into a 3.7.0 regression of CPython failing to build them, that I reported and seems to be valid bug of theirs.
So they will be missing for some longer time. Actually I wasn't so sure if
they are all that useful, or working as expected for the runners, but with
-m nuitka mode of execution, that ought to be a non-issue. so it
would be nice to keep them for those who use them for deployment internally.
I have a change here. This is going to be a draft post until I publish it, so I might the link, or mention it on the list, but I do not think I will wait for feedback, where there is not going to be all that much.
So I am shooting this off the web site.
This is an exciting field of work, that I have been busy with this week. I will briefly describe the issue at hand.
So generators in Python are more generally called coroutines in other places, and basically that is code shaking hands, executing resuming in one, handing back a piece of data back and forth.
In Python, the way of doing this is
yield and more recently
as a convienant way for of doing it in a loop in Python3. I still recall the
days when that was a statement. Then communication was one way only. Actually
when I was still privately doing Nuitka based on then Python 2.5 and was then
puzzled for Python 2.6, when I learned in Nuitka about it becoming an expression.
The way this is implemented in Python, is that execution of a frame is simply
suspended, and another frame stack bytecode is activated. This switching is of
course very fast potentially, the state is already fully preserved on the stack
of the virtual machine, which is owned by the frame. For Nuitka, when it still
was C++, it wasn't going to be possible to interrupt execution without preserving
the stack. So what I did was very similar, and I started to use
makecontext/setcontext to implement what I call fibers.
Basically that is C level stack switching, but with a huge issue. Python does not grow stacks, but can need a lot of stack space below. Therefore 1MB or even 2MB per generator was allocated, to be able to make deep function calls if needed.
So using a lot of generators on 32 bits could easily hit a 2GB limit. And now with Python3.5 coroutines people use more and more of them, and hit memory issues.
So, goto generators, now that C is possible, are an entirely new solution. With it, Nuitka will use one stack only. Generator code will become re-entrant, store values between entries on the heap, and continue execution at goto destinations dispatched by a switch according to last exit of the generator.
So I am now making changes to cleanup the way variable declarations and accesses for the C variables are being made. More on that next week though. For now I am very exited about the many cleanups that stem from it. The code generation used to have many special bells and whistles, and they were generalized into one thing now, making for cleaner and easier to understand Nuitka code.
On interesting thing, is that an incompatibility related to
__new__ will go
staticmethod that we had to hack into it, because the
Python core will do it for uncompiled functions only, had to be done while
declaring the class. So it was visible and causing issues with at least the
Python enum module, which wants to call your
__new__ manually. Because
why would it not?!
But turns out, for Python3 the
staticmethod is not needed anymore. So
this is now only done for Python2, where it is needed, and things work
smoothly with this kind of code now too. This is currently in my factory
testing and will probably become part of a hotfix if it turns out good.
Immediately after the release, some rarely run test, where I compiled all the code on my machine, found 2 older bugs, obscure ones arguably, that I made into a hotfix, also because the test runner was having a regression with 3.7, which prevented some package builds. So that was 0.5.32.1 release.
And then I received a bug report about
await where a self test of Nuitka
fails and reports an optimization error. Very nice, the new exceptions that
automatically dump involved nodes as XML made it immediately clear from the
report, what is going on, even without having to reproduce anything. I bundled
a 3.7 improvement for error cases in class creation with it. So that was the
Finishing goto generators is my top priority, but I am also going over minor issues with the 3.7 test suite, fixing test cases there, and as with e.g. the enum issue, even known issues this now finds.
Until next week.