Nuitka Release 0.6.16

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is mostly polishing and new features. Optimization looked only at threading performance, and LTO improvements on Windows.

Bug Fixes

  • Fix, the pkg-resources failed to resolve versions for importlib.metadata from its standard library at compile time. Fixed in 0.6.15.1 already.

  • Standalone: Fix, --include-module was not including the module if it was an extension modules, but only for Python modules. Fixed in 0.6.15.1 already.

  • Standalone: Added missing implicit dependencies for gi.overrides. Fixed in 0.6.15.1 already.

  • Python3.9: Fix, could crash when using generic aliases in certain configurations. Fixed in 0.6.15.2 already.

  • Fix, the tensorflow plugin needed an update due to changed API. Fixed in 0.6.15.3 already.

  • When error exiting Nuitka, it now closes any open progress bar for cleaner display.

  • Standalone: Added missing dependency for skimage.

  • Standalone: The numpy plugin now automatically includes Qt backend if any of the Qt binding plugins is active.

New Features

  • Pyton3.5+: Added support for onefile compression. This is using zstd which is known to give very good compression with very high decompression, much better than e.g. zlib.

  • macOS: Added onefile support.

  • FreeBSD: Added onefile support.

  • Linux: Added method to use tempdir onefile support as used on other platforms as an alternative to AppImage based.

  • Added support for recursive addition of files from directories with patterns.

  • Attaching the payload to onefile now has a progress bar too.

  • Windows: Prelimary support for the yet unfinished Nuitka-Python that allows static linking and higher performance on Windows, esp. with Nuitka.

  • Windows: In acceleration mode, for uninstalled Python, now a CMD file is created rather than copying the DLL to the binary directory. That avoids conflicts with architectures and of course useless file copies.

  • New abilities for plugin anti-bloat allow to make it an error when certain modules are imported. Added more specific options for usual trouble makes, esp. setuptools, pytest are causing an explosion for some programs, while being unused code. This makes it now easier to oversee this.

  • It's now possible to override appdirs decision for where cache files live with an environment variable NUITKA_CACHE_DIR.

  • The -o option now also works with onefile mode, it previously rejected anything but acceleration mode. Fixed in 0.6.15.3 already.

  • Plugins: It's now possible for multiple plugins to provide pre or post load code for the same module.

  • Added indications for compilation modes standalone and onefile to the __compiled__ attribute.

  • Plugins: Give nicer error message in case of colliding command line options.

Optimization

  • Faster threading code is now using for Python3.8 or higher and not only 3.9, giving a performance boost, esp. on Windows.

  • Using --lto is now the default with MSVC 2019 or higher. This will given smaller and faster binaries. It has been available for some time, but not been the default yet.

Cleanups

  • Using different progress bar titles for C compilation of Python code and C compilation of onefile bootstrap.

  • Moved platform specific detections, for FreeBSD/OpenBSD/macOS out of the Scons file and to common Nuitka code, sometimes eliminating duplications with one version being more correct than the other.

  • Massive cleanup of datafile plugin, using pattern descriptions, so more code duplication can be removed.

  • More cleanup of the scons files, sharing more common code.

Organisational

  • Under the name Nuitka-Python we are now also developing a fork of CPython with enhancements, you can follow and joint it at https://github.com/Nuitka/Nuitka-Python but at this time it is not yet ready for prime time.

  • Onefile under Windows now only is temporary file mode. Until we figure out how to solve the problems with locking and caching, the mode where it installs to the AppData of the user is no longer available.

  • Renamed the plugin responsible for PyQt5 support to match the names of others. Note however, that at this time, PySide2 or PySide6 are to be recommended.

  • Make it clear that PySide 6.1.2 is actually going to be the supported version of PySide6.

  • Use MSVC in Github actions.

Summary

This release had a massive focus on expanding existing features, esp. for onefile, and plugins API, such that we can now configure anti-bloat with yaml, have really nice datafile handling options, and have onefile on all OSes practically.

Nuitka Release 0.6.15

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release polished previous work with bug fixes, but there are also important new things that help make Nuitka more usable, with one important performance improvement.

Bug Fixes

  • Fix, hard imports were not automatically used in code generation leading to errors when used. Fixed in 0.6.14.2 already.

  • Windows: Fix, clcache was disabled by mistake. Fixed in 0.6.14.2 already.

  • Standalone: Added data files for jsonschema to be copied automatically.

  • Standalone: Support for pendulum wasn't working anymore with the last release due to plugin interface issues.

  • Retry downloads without SSL if that fails, as some Python do not have working SSL. Fixed in 0.6.14.5 already.

  • Fix, the ccache path wasn't working if it contained spaces. Fixed in 0.6.14.5 already.

  • Onefile: For Linux and ARM using proper download off appimage. Fixed in 0.6.14.5 already.

  • Standalone: Added support for pyreadstat. Fixed in 0.6.14.5 already.

  • Standalone: Added missing dependencies for pandas. Fixed in 0.6.14.6 already.

  • Standalone: Some preloaded packages from .pth do not have a __path__, these can and must be ignored.

  • Onefile: On Linux, the sys.argv[0] was not the original file as advertised.

  • Standalone: Do not consider .mesh and .frag files as DLls in the Qt bindings when including the qml support. This was causing errors on Linux, but was generally wasteful.

  • Fix, project options could be injected twice, which could lead to errors with options that were only allowed once, e.g. --linux-onefile-icon.

  • Windows: When updating the resources in created binaries, treat all kinds of OSError with information output.

  • Onefile: Remove onefile target binary path at startup as well, so it cannot cause confusion after error exit.

  • Onefile: In case of error exit from AppImage, preserve its outputs and attempt to detect if there was a locking issue.

  • Standalone: Scan package folders on Linux for DLLs too. This is necessary to properly handle PyQt5 in case of Qt installed in the system as well.

  • Standalone: On Linux, standard QML files were not found.

  • Standalone: Enforce C locale when detecting DLLs on Linux, otherwise whitelisting messages didn't work properly on newer Linux.

  • Standalone: Added support for tzdata package data files.

  • Standalone: Added support for exchangelib.

  • Python3.9: Fix, type subscripts could cause optimization errors.

  • UI: Project options didn't properly handle quoting of arguments, these are normally removed by the shell.

  • Linux: The default icon for Python in the system is now found with more version specific names and should work on more systems.

  • Standalone: Do not include libstdc++ as it should come from the system rather.

New Features

  • Added plugin anti-bloat plugin, intended to fight bloat. For now it can make including certain modules an error, a warning, or force ImportError, e.g. --noinclude-setuptools-mode=nofollow is very much recommended to limit compilation size.

  • The pkg-resources builtin now covers metadata and importlib_metadata packages for compile time version resolution as well.

  • Added support for PySide2 on Python version before 3.6, because the patched code needs no workarounds. Fixed in 0.6.14.3 already.

  • Windows: Convert images to icon files on the fly. So now you can specify multiple PNG files, and Nuitka will create an icon out of that automatically.

  • macOS: Automatically download ccache binary if not present.

  • Plugins: New interface to query the main script path. This allows plugins to look at its directory.

  • UI: Output the versions of Nuitka and Python during compilation.

  • UI: Added option to control static linking. So far this had been enabled only automatically for cases where we are certain, but this allows to force enable or disable it. Now an info is given, if Nuitka thinks it might be possible to enable it, but doesn't do it automatically.

  • UI: Added --no-onefile to disable --onefile from project options.

Optimization

  • Much enhanced GIL interaction with Python3.9 giving a big speed boost and better threading behaviour.

  • Faster conversion of iterables to list, if size can be know, allocation ahead saves a lot of effort.

  • Added support for GenericAlias objects as compile time constants.

Organisational

  • Enhanced Github issue raising instructions.

  • Apply rstfmt to all documentation and make it part of the commit hook.

  • Make sure to check Scons files as well. This would have caught the code used to disable clcache temporarily.

  • Do not mention Travis in PR template anymore, we have stopped using it.

  • Updated requirements to latest versions.

  • Bump requirements for development to 3.7 at least, toosl like black do not work with 3.6 anymore.

  • Started work on Nuitka Python, a CPython fork intended for enhanced performance and standalone support with Nuitka.

Cleanups

  • Determine system prefix without virtualenv outside of Scons, such that plugins can share the code. There was duplication with the numpy plugin, and this will only be more complete using all approaches. This also removes a lot of noise from the scons file moving it to shared code.

  • The Qt plugins now collect QML files with cleaner code.

Tests

  • Nicer error message if a wrong search mode is given.

  • Windows: Added timeout for determining run time traces with dependency walker, sometimes this hangs.

  • Added test to cover the zip importer.

  • Making use of project options in onefile tests, making it easier to execute them manually.

Summary

For Windows, it's now easier than ever to create an icon for your deployment, because you can use PNG files, and need not produce ICO files anymore, with Nuitka doing that for you.

The onefile for Linux had some more or less severe problems that got addressed, esp. also when it came to QML applications with PySide.

On the side, we are preparing to greatly improve the caching of Nuitka, starting with retaining modules that were demoted to bytecode. There are changes in this release, to support that, but it's not yet complete. We expect that scalability will then be possible to improve even further.

Generally this is mostly a maintenance release, which outside of the threading performance improvement has very little to offer for faster execution, but that actually does a lot. Unfortunately right now it's limited to 3.9, but some of the newer Python's will also be supported in later releases.

Nuitka Release 0.6.14

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release has few, but important bug fixes. The main focus was on expanding standalone support, esp. for PySide2, but also and in general with plugins added that workaround pkg_resources usage for version information.

Also an important new features was added, e.g. the project configuration in the main file should prove to be very useful.

Bug Fixes

  • Compatibility: Fix, modules that failed to import, should be retried on next import.

    So far we only ever executed the module body once, but that is not how it's supposed to be. Instead, only if it's in sys.modules that should happen, which is the case after successful import.

  • Compatibility: Fix, constant False values in right hand side of and/or conditions were generating wrong code if the left side was of known bool shape too.

  • Standalone: Fix, add styles Qt plugins to list of sensible plugins.

    Otherwise no mouse hover events are generated on some platforms.

  • Compatibility: Fix, relative from imports beyond level 1 were not loadingg modules from packages if necessary. Fixed in 0.6.13.3 already.

  • Standalone: The crypto DLL check for Qt bindings was wrong. Fixed in 0.6.13.2 already.

  • Standalone: Added experimental support for PySide6, but for good results, 6.1 will be needed.

  • Standalone: Added support for newer matplotlib. Fixed in 0.6.12.1 already.

  • Standalone: Reverted changes related to pkg_resources that were causing regressions. Fixed in 0.6.13.1 already.

  • Standalone: Adding missing implicit dependency for cytoolz package. Fixed in 0.6.13.1 already.

  • Standalone: Matching for package names to not suggest recompile for was broken and didn't match. Fixed in 0.6.13.1 already.

New Features

  • Added support for project options.

    When found in the filename provided, Nuitka will inject options to the commandline, such that it becomes possible to do a complex project with only using

    python -m nuitka filename.py
    
    # Compilation mode, support OS specific.
    # nuitka-project-if: {OS} in ("Windows", "Linux"):
    #    nuitka-project: --onefile
    # nuitka-project-if: {OS} not in ("Windows", "Linux"):
    #    nuitka-project: --standalone
    
    # The PySide2 plugin covers qt-plugins
    # nuitka-project: --enable-plugin=pyside2
    # nuitka-project: --include-qt-plugins=sensible,qml
    
    # The pkg-resources plugin is not yet automatic
    # nuitka-project: --enable-plugin=pkg-resources
    
    # Nuitka Commercial only features follow:
    
    # Protect the constants from being readable.
    # nuitka-project: --enable-plugin=data-hiding
    
    # Include datafiles for Qt into the binary directory.
    # nuitka-project: --enable-plugin=datafile-inclusion
    # nuitka-project: --qt-datadir={MAIN_DIRECTORY}
    # nuitka-project: --qt-datafile-pattern=*.js
    # nuitka-project: --qt-datafile-pattern=*.qml
    # nuitka-project: --qt-datafile-pattern=*.svg
    # nuitka-project: --qt-datafile-pattern=*.png
    

    Refer to the user manual for a table of directives and the variables allowed to be used.

  • Added option to include whole data directory structures in standalone.

    The new option --include-data-dir was added and is mostly required for onefile mode, but recommended for standalone too.

  • Added pkg-resources plugin.

    This one can resolve code like this at compile time without any need for pip metadata to be present or used.

    pkg_resources.get_distribution("module_name").version
    pkg_resources.get_distribution("module_name").parsed_version
    
  • Standalone: Also process early imports in optimization.

    Otherwise plugins cannot work on standard library modules. This makes it possible to handle them as well.

Optimization

  • Faster binary operations.

    Applying lessons learnt during the enhancements for in-place operations that initially gave worse results than some manual code, we apply the same tricks for all binary operations, which speeds them up by significant margins, e.g. 30% for float addition, 25% for Python int addition, and still 6% for Python int addition.

  • More direct optimization of unary operations on constant value.

    Without this, -1 was not directly a constant value, but had to go through the unary - operation, which it still does, but now it's done at tree building time.

  • More direct optimization for not in branches.

    Invertible comparisons, i.e. is/is not and in/not in do not have do be done during optimization. This mainly avoids noise during optimization from such unimportant steps.

  • More direct optimization for constant slices.

    These are used in Python3 for all subscripts, e.g. a[1:2] will use slice(1,2) effectively. For Python2 they are used less often, but still. This also avoids a lot of noise during optimization, mostly on Python3

  • Scons: Avoid writing database to disk entirely.

    This saves a bit of disk churn and makes it unnecessary to specify the location such that it doesn't collide between Python versions.

  • For optimization passes, use previous max total as minimum for next pass. That will usually be a more accurate result, rather than starting from 1 again. Part of 0.6.13.1 already.

  • Enhancements to the branch merging improve the scalability of Nuitka somewhat, although the merging itself is still not very scalable, there are some modules that are very slow to optimize still.

  • Use orderset if available over the inline copy for OrderedSet which is much faster and improves Nuitka compile times.

  • Make pkgutil a hard import too, this is in preparation of more optimization for its functions.

Organisational

  • Upstream patches for PySide6 have been contributed and merged into the development branch dev. Full support should be available once this is released as part of 6.1 which is waiting for Qt 6.1 naturally.

  • Patches for PySide2 are available to commercial customers, see PySide2 support page.

  • Formatted all documents with rstfmt and made that part of the commit hook for Nuitka. It now works for all documents we have.

  • Updated inline copy of tqdm to 4.59.0 which ought to address spurious errors given.

  • User Manual: Remove --show-progress from the tutoral. The default progress bar is then disabled, and is actually much nicer to use.

  • Developer Manual: Added description of how context managers should be named.

  • Cleanup language for some warnings and outputs.

    It was still using obsolete "recursion" language rather than talking about "following imports", which is the new one.

Cleanups

  • Remove dead code related to constants marshal, the data composer has replaced this.

  • Avoid internal API usage for loading extension modules on Linux, there is a function in sys module to get the ld flags.

Tests

  • Fix, the only mode wasn't working properly.

  • Use new project options feature for specific options in basic tests allowing to remove them from the test runner.

Summary

For PySide2 things became more perfect, but it takes upstream patches unfortunately such that only PySide6.1 will be working out of the box outside of the commercial offering. We will also attempt to provide workarounds, but there are some things that cannot be done that way.

This release added some more scalability to the optimization process, however there will be more work needed to make efficient branch merges.

For onefile, a feature to include whole directories had been missing, and could not easily be achieved with the existing options. This further rounds this up, now what's considered missing is compression and macOS support, both of which should be coming in a future release.

For the performance side of things, the binary operator work can actually yield pretty good gains, with double digit improvements, but this covers only so much. Much more C types and better type tracing would be needed, but there was no progress on this front. Future releases will have to revisit the type tracing to make sure, we know more about loop variables, etc. so we can achieve the near C speed we are looking for, at least in the field of int performance.

This release has largely been driven by the Nuitka Commercial offering and needs for compatibility with more code, which is of course always a good thing.

Nuitka Release 0.6.13

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release follows up with yet again massive improvement in many ways with lots of bug fixes and new features.

Bug Fixes

  • Windows: Icon group entries were not still not working properly in some cases, leading to no icon or too small icons being displayed. Fixed in 0.6.12.2 already.

  • Windows: Icons and version information were copied from the standalone executable to the onefile executable, but that failed due to race situations, sometimes reproducible. Instead we now apply things to both independently. Fixed in 0.6.12.2 already.

  • Standalone: Fixup scanning for DLLs with ldconfig on Linux and newer versions making unexpected outputs. Fixed in 0.6.12.2 already.

  • UI: When there is no standard input provided, prompts were crashing with EOFError when --assume-yes-for-downloads is not given, change that to defaulting to "no" instead. Fixed in 0.6.12.2 already.

  • Windows: Detect empty strings for company name, product name, product and file versions rather than crashing on them later. Them being empty rather than not there can cause a lot of issues in other places. Fixed in 0.6.12.2 already.

  • Scons: Pass on exceptions during execution in worker threads and abort compilation immediately. Fixed in 0.6.12.2 already.

  • Python3.9: Still not fully compatible with typing subclasses, the enhanced check is now closely matching the CPython code. Fixed in 0.6.12.2 already.

  • Linux: Nicer error message for missing libfuse requirement.

  • Compatibility: Lookups on dictionaries with None value giving a KeyError exception, but with no value, which is not what CPython does.

  • Python3.9: Fix, future annotations were crashing in debug mode. Fixed in 0.6.12.3 already.

  • Standalone: Corrections to the glfw were made. Fixed in 0.6.12.3 already.

  • Standalone: Added missing ìmplicit dependency for py.test. Fixed in 0.6.12.3 already.

  • Standalone: Adding missing implicit dependency for pyreadstat.

  • Windows: Added workaround for common clcache locking problems. Since we use it only inside a single Scons process, we can avoiding using Windows mutexes, and use a process level lock instead.

  • Plugins: Fix plugin for support for eventlet. Fixed in 0.6.12.3 already.

  • Standalone: Added support for latest zmq on Windows.

  • Scons: the --quiet flag was not fully honored yet, with Scons still making a few outputs.

  • Standalone: Added support for alternative DLL name for newer PyGTK3 on Windows. Fixed in 0.6.12.4 already.

  • Plugins: Fix plugin for support for gevent. Fixed in 0.6.12.4 already.

  • Standalone: Added yet another missing implicit dependency for pandas.

  • Plugins: Fix, the qt-plugins plugin could stumble over .mesh files.

  • Windows: Fix, dependency walker wasn't properly working with unicode %PATH% which could e.g. happen with a virtualenv in a home directory that requires them.

  • Python3: Fixed a few Python debug mode warnings about unclosed files that have sneaked into the codebase.

New Features

  • Added new options --windows-force-stdout-spec and --windows-force-stderr-spec to force output to files. The paths provided at compile time can resolve symbolic paths, and are intended to e.g. place these files near the executable. Check the User Manual for a table of the currently supported values. At this time, the feature is limited to Windows, where the need arose first, but it will be ported to other supported OSes eventually. These are most useful for programs run as --windows-disable-console or with --plugin-enable=windows-service.

  • Windows: Added option --windows-onefile-tempdir-spec to provide the temporary directory used with --windows-onefile-tempdir in onefile mode, you can now select your own pattern, and e.g. hardcode a base directory of your choice rather than %TEMP.

  • Added experimental support for PySide2 with workarounds for compiled methods not being accepted by its core. There are known issues with PySide2 still, but it's working fine for some people now. Upstream patches will have to be created to remove the need for workarounds and full support.

Optimization

  • Use binary operation code for their in-place variants too, giving substantial performance improvements in all cases that were not dealt with manually already, but were covered in standard binary operations. Until now only some string, some float operations were caught sped up, most often due to findings of Nuitka being terribly slower, e.g. not reusing string memory for inplace concatenation, but now all operations have code that avoids a generic code path, that is also very slow on Windows due calling to using the embedded Python via API being slow.

  • For mixed type operations, there was only one direction provided, which caused fallbacks to slower forms, e.g. with long and float values leading to inconsistent results, such that a - 1 and 1 - a being accelerated or not.

  • Added C boolean optimization for a few operations that didn't have it, as these allow to avoid doing full computation of what the object result would have to do. This is not exhausted fully yet.

  • Python3: Faster +/-/+=/-= binary and in-place operations with int values providing specialized code helpers that are much faster, esp. in cases where no new storage is allocated for in-place results and where not a lot of digits are involved.

  • Python2: The Python3 int code is the Python2 long type and benefits from the optimization of +/-/+=/-= code as well, but of course its use is relatively rare.

  • Improved __future__ imports to become hard imports, so more efficient code is generated for them.

  • Counting of instances had a runtime impact by providing a __del__ that was still needed to be executed and limits garbage collection on types with older Python versions.

  • UI: Avoid loading tqdm module before it's actually used if at all (it may get disabled by the user), speeding up the start of Nuitka.

  • Make sure to optimize internal helpers only once and immediately, avoiding extra global passes that were slowing down Python level compilation by of large programs by a lot.

  • Make sure to recognize the case where a module optimization can provide no immediate change, but only after a next run, avoiding extra global passes originating from these, that were slowing down compilation of large programs by a lot. Together with the other change, this can improve scalability by a lot.

  • Plugins: Remove implicit dependencies for pkg_resources.extern and use aliases instead. Using one of the packages, was causing all that might be used, to be considered as used, with some being relatively large. This was kind of a mistake in how we supported this so far.

  • Plugins: Revamped the eventlet plugin, include needed DNS modules as bytecode rather than as source code, scanning them with pkgutil rather than filesystem, with much cleaner code in the plugin.

Organisational

  • Removed support for pefile dependency walker choice and inline copy of the code. It was never as good giving incomplete results, and after later improvements, slower, and therefore has lost the original benefit over using Dependency Walker that is faster and more correct.

  • Added example for onefile on Windows with the version information and with the temporary directory mode.

  • Describe difference in file access with onefile on Windows, where sys.argv[0] and os.path.dirname(__file__) will be different things.

  • Added inline copy of tqdm to make sure it's available for progress bar output for 2.7 or higher. Recommend having it in the Debian package.

  • Added inline copy of colorama for use on Windows, where on some terminals it will give better results with the progress bar.

  • Stop using old PyLint for Python2, while it would be nice to catch errors, the burden of false alarms seems to high now.

  • UI: Added even more checks on options that make no sense, made sure to do this only after a possible restart in proper environment, so warnings are not duplicated.

  • For Linux onefile, keep appimage outputs in case of an error, that should help debugging it in case of issues.

  • UI: Added traces for plugin provided implicit dependencies leading to inclusions.

  • Added inline copy of zstd C code for use in decompression for the Windows onefile bootstrap, not yet used though.

  • Added checks to options that accept package names for obvious mistakes, such that --include-package-data --mingw64 will not swallow an option, as that is clearly not a package name, that would hide that option, while also not having any intended effect.

  • Added ignore list for decision to recompile extension modules with available source too. For now, Nuitka will not propose to recompile Cython modules that are very likely not used by the program anyway, and also not for lxml until it's clear if there's any benefit in that. More will be added in the future, this is mostly for cases, where Cython causes incompatibilities.

  • Added support for using abstract base classes in plugins. These are not considered for loading, and allow nicer implementation of shared code, e.g. between PyQt5 and PySide2 plugins, but allow e.g. to enforce the provision of certain overloads.

  • User Manual: Remove the instruction to install clcache, since it's an inline copy, this makes no sense anymore and that was obsolete.

  • Updated PyLint to latest versions, and our requirements in general.

Cleanups

  • Started removal of PyLint annotations used for old Python2 only. This will be a continuous action to remove these.

  • Jinja2 based static code generation for operations was cleaned up, to avoid code for static mismatches in the result C, avoiding language constructs like if (1 && 0) with sometimes larger branches, replacing it with Jinja2 branches of the {% if ... %} form.

  • Jinja2 based Python2 int code was pioniering the use of macros, but this was expanded to allow kinds of types for binary operations, allow their reuse for in-place operation, with these macros making returns via goto exits rather than return statements in a function. Landing pads for these exits can then assign target values for in-place different from what those for binary operation result return do.

  • Changed the interfacing of plugins with DLL dependency detection, cleaning up the interactions considerably with more unified code, and faster executing due to cached plugin decisons.

  • Integrate manually provided slot function for unicode and str into the standard static code generation. Previously parts were generated and parts could be generated, but also provided with manual code. The later is now all gone.

  • Use a less verbose progress bar format with less useless infos, making it less likely to overflow.

  • Cleanup how payload data is accessed in Windows onefile bootstrap, preparing the addition of decompression, doing the reading from the file in only one dedicated function.

  • When Jinja2 generated exceptions in the static code, it is now done via proper Jinja2 macros rather than Python code, and these now avoid useless Python version branches where possible, e.g. because a type like bytes is already Python version specific, with the goal to get rid of PyErr_Format usage in our generated static code. That goal is future work though.

  • Move safe strings helpers (cannot overflow) to a dedicated file, and remove the partial duplication on the Windows onefile bootstrap code.

  • The Jinja2 static code generation was enhanced to track the usage of labels used as goto targets, so that error exits, and value typed exits from operations code no longer emitted when not used, and therefore labels that are not used are not present.

  • For implicit dependencies, the parsing of the .pyi file of a module no longer emits a dependency on the module itself. Also from plugins, these are now filtered away.

Tests

  • Detect if onefile mode has required downloads and if there is user consent, otherwise skip onefile tests in the test runner.

  • Need to also allow accesses to files via short paths on Windows if these are allowed long paths.

  • The standalone tests on Windows didn't actually take run time traces and therefore were ineffective.

  • Added standalone test for glfw coverage.

  • Construct based tests for in-place operations are now using a value for the first time, and then a couple more times, allowing for real in-place usage, so we are sure we measure correctly if that's happening.

Summary

Where the big change of the last release were optimization changes to reduce the global passes, this release addresses remaining causes for extra passes, that could cause these to still happen. That makes sure, Nuitka scalability is very much enhanced in this field again.

The new features for forced outputs are at this time Windows only and make a huge difference when it comes to providing a way to debug Windows Services or programs in general without a console, i.e. a GUI program. These will need even more specifiers, e.g. to cover program directory, rather than exe filename only, but it's a very good start.

On the tooling side, not a lot has happened, with the clcache fix, it seems that locking issues on Windows are gone.

The plugin changes from previous releases had left a few of them in a state where they were not working, but this should be restored. Interaction with the plugins is being refined constantly, and this releases improved again on their interfaces. It will be a while until this becomes stable.

Adding support for PySide2 is a headline feature actually, but not as perfect as we are used to in other fields. More work will be needed, also in part with upstream changes, to get this to be fully supported.

For the performance side of things, the in-place work and the binary operations work has taken proof of concept stuff done for Python2 and applied it more universally to Python3. Until we cover all long operations, esp. * seems extremely important and is lacking, this cannot be considered complete, but it gives amazing speedups in some cases now.

Future releases will revisit the type tracing to make sure, we know more about loop variables, to apply specific code helpers more often, so we can achieve the near C speed we are looking for in the field of int performance.

Nuitka Release 0.6.12

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is yet again a massive improvement in many ways with lots of bug fixes and new features.

Bug Fixes

  • Windows: Icon group entries were not working properly in some cases, leading to no icon or too small icons being displayed.

  • Standalone: The PyQt implicit dependencies were broken. Fixed in 0.6.11.1 already.

  • Standalone: The datafile collector plugin was broken. Fixed in 0.6.11.3 already.

  • Standalone: Added support for newer forms of matplotlib which need a different file layout and config file format. Fixed in 0.6.11.1 already.

  • Plugins: If there was an error during loading of the module or plugin, it could still be attempted for use. Fixed in 0.6.11.1 already.

  • Disable notes given by gcc, these were treated as errors. Fixed in 0.6.11.1 already.

  • Windows: Fix, spaces in gcc installation paths were not working. Partially fixed in 0.6.11.4 already.

  • Linux: Fix, missing onefile icon error message was not complete. Fixed in 0.6.11.4 already.

  • Standalone: Workaround zmq problem on Windows by duplicating a DLL in both expected places. Fixed in 0.6.11.4 already.

  • Standalone: The dill-compat plugin wasn't working anymore. Fixed in 0.6.11.4 already.

  • Windows: Fix mistaken usage of sizeof for wide character buffers. This caused Windows onefile mode in temporary directory. Fixed in 0.6.11.4 already.

  • Windows: Fix, checking subfolder natured crashed with different drives on Windows. Fixed in 0.6.11.4 already.

  • Windows: Fix, usage from MSVC prompt was no longer working, detect used SDK properly. Fixed in 0.6.11.4 already.

  • Windows: Fix, old clcache installation uses pth files that prevented our inline copy from working, workaround was added.

  • Windows: Also specify stack size to be used when compiling with gcc or clang.

  • Fix, claim of Python 3.9 support also in PyPI metadata was missing. Fixed in 0.6.11.5 already.

  • Python3.9: Subscripting type for annotations wasn't yet implemented.

  • Python3.9: Better matching of types for metaclass selection, generic aliases were not yet working, breaking some forms of type annotations in base classes.

  • Windows: Allow selecting --msvc-version when a MSVC prompt is currently activated.

  • Windows: Do not fallback to using gcc when --msvc-version has been specified. Instead it's an error if that fails to work.

  • Python3.6+: Added support for del () statements, these have no effect, but were crashing Nuitka.

    del a  # standard form
    del a, b  # same as del a; del b
    del (a, b)  # braces are allowed
    del ()  # allowed for consistency, but wasn't working.
    
  • Standalone: Added support for glfw through a dedicated plugin.

  • macOS: Added support for Python3 from system and CPython official download for latest OS version.

New Features

  • UI: With tqdm installed alongside Nuitka, experimental progress bars are enabled. Do not use `` --show-progress`` or --verbose as these might have to disable it.

  • Plugins: Added APIs for final processing of the result and onefile post processing.

  • Onefile: On Windows, the Python process terminates with KeyboardInterrupt when the user sends CTRL-break, CTRL-C, shutdown or logoff signals.

  • Onefile: On Windows, in case of the launching process terminating unexpectedly, e.g. due to Taskmanager killing it, or a os.sigkill resulting in that, the Python process still terminates with KeyboardInterrupt.

  • Windows: Now can select icons by index from files with multiple icons.

Optimization

  • Avoid global passes caused by module specific optimization. The variable completeness os now traced per module and function scope, allowing a sooner usage. Unused temporary variables and closure variables are remove immediately. Recognizing possible auto releases of parameter variables is also instantly.

    This should bring down current passes from 5-6 global passes to only 2 global passes in the normal case, reducing frontend compile times in some cases massively.

  • Better unary node handling. Dedicated nodes per operation allow for more compact memory usage and faster optimization.

  • Detect flow control and value escape for the repr of node based on type shape.

  • Enhanced optimization of caught exception references, these never raise or have escapes of control flow.

  • Exception matching operations are more accurately annotated, and may be recognized to not raise in more cases.

  • Added optimization for the issubclass built-in.

  • Removed scons caching as used on Windows entirely. We should either be using clcache or ccache automatically now.

  • Make sure to use __slots__ for all node classes. In some cases, mixins were preventing the feature from being it. We now enforce their correct specification of slots, which makes sure we can't miss it anymore. This should again gain more speed and save memory at frontend compile time.

  • Scons: Enhanced gcc version detection with improved caching behavior, this avoids querying the same gcc binary twice.

Organisational

  • The description of Nuitka on PyPI was absent for a while. Added back by adding long description of the project derived from the README file.

  • Avoid terms blacklist, whilelist and slave in the Nuitka code preferring blocklist, ignorelist and child instead, which are actually more clear anyway. We follow a general trend to do this.

  • Configured the inline copy of Scons so pylance has an easier time to find it.

  • The git commit hook had stopped applying diffs with newest git, improved that.

  • Updated description for adding new CPython test suite.

  • Using https URLs for downloading dependency walker, for it to be more secure.

  • The commit hook can now be disabled, it's in the Developer Manual how to do it.

Cleanups

  • Moved unary operations to their own module, the operators module was getting too crowded.

  • The scons files for Python C backend and Windows onefile got cleaned up some more and moved more common code to shared modules.

  • When calling external tools, make sure to provide null input where possible.

  • Unified calling install_name_tool into a single method for adding rpath and name changes both at the same time.

  • Unified how tools like readelf, ldconfig etc. are called and error exits and outputs checked to make sure we don't miss anything as easily.

Tests

  • Adapted for some openSUSE specific path usages in standalone tests.

  • Basic tests for onefile operation and with termination signal sent were added.

Summary

The big changes in this release are the optimization changes to reduce the global passes and the memory savings from other optimization. These should again make Nuitka more scalable with large projects, but there definitely is work remaining.

Adding nice stopping behaviour for the Onefile mode on Windows is seemingly a first, and it wasn't easy, but it will make it more reliable to users.

Also tooling of gcc and MSVC on Windows got a lot more robust, covering more cases, and macOS support has been renewed and should be a lot better now.

The progress bar is a nice touch and improves the overall feel of the compilation process, but obviously we need to aim at getting faster overall still. For projects using large dependencies, e.g. Pandas the compilation is still far too slow and that will need work on caching frontend results, and better optimization and C code generation for the backend.

Nuitka Release 0.6.11

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release is a massive improvement in many ways with lots of bug fixes and new features.

Bug Fixes

  • Fix, the .pyi file parser didn't handle relative imports. Fixed in 0.6.10.1 already.

  • Windows: Fix, multiprocessing plugin was not working reliable following of imports from the additional entry point. Fixed in 0.6.10.1 already.

  • Pipenv: Workaround parsing issue with our setup.py to allow installation from Github. Fixed in 0.6.10.1 already.

  • Merging of branches in optimization could give indetermistic results leading to more iterations than necessary. Fixed in 0.6.10.1 already.

  • Windows: Avoid profile powershell when attempting to resolve symlinks. Fixed in 0.6.10.1 already.

  • Windows: Fix, always check for stdin, stdout, and stderr presence. This was so far restricted to gui mode applications, but it seems to be necessary in other situations too. Fixed in 0.6.10.1 already.

  • Python2: Fix, --trace-execution was not working for standalone mode but can be useful for debugging. Fixed in 0.6.10.1 already.

  • Windows: Onefile could run into path length limits. Fixed in 0.6.10.3 already.

  • Windows: The winlib gcc download link became broken and was updated. Fixed in 0.6.10.3 already.

  • Plugins: The "__main__" module was not triggering all plugin hooks, but it needs to for completeness.

  • Standalone: Fix, symlinked Python installations on Windows were not working, with dependency walker being unable to look into these. Fixed in 0.6.10.4 already.

  • Standalone: Fix support for numpy on Windows and macOS, the plugin failed to copy important DLLs. Fixed in 0.6.10.4 already.

  • Python3: For versions before 3.7, the symlink resolution also needs to be done, but wasn't handling the bytes output yet. Fixed in 0.6.10.4 already.

  • Fix, folder based inclusion would both pick up namespace folders and modules of the same name, crashing the compilation due to conflicts. Fixed in 0.6.10.4 already.

  • Fix, the --lto wasn't used for clang on non-Windows yet.

  • Fix, the order of locals dict releases wasn't enforced, which could lead to differences that break caching of C files potentially. Fixed in 0.6.10.5 already.

  • Fix, hash nodes didn't consider if their argument was raising, even if the type of the argument was str and therefore the operation should not. Fixed in 0.6.10.5 already.

  • Fix, need to copy type shape and escape description for the replacement inverted comparisons when used with not, otherwise the compilation can crash as these are expected to be present at all times. Fixed in 0.6.10.5 already.

  • Fix, some complex constant values could be confused, e.g. -0j and 0j. These corner cases were not properly considered in the constant loading code, only for float so far.

  • Standalone: Fix, bytecode only standard library modules were not working. This is at least used with Fedora 33.

  • Linux: Fix, extension modules compiled with --lto were not working.

  • Windows: Retry if updating resources fails due to Virus checkers keeping files locked.

  • Plugins: Pre- and postload code of modules should not be allowed to cause ImportError, as these will be invisible to the other parts of optimization, instead make them unraisable error traces.

  • Standalone: Adding missing import for SciPy 1.6 support.

  • Windows: Fix, only export required symbols when using MinGW64 in module mode.

New Features

  • Python3.9: Added official support for this version.

  • Onefile: Added command line options to include data files. These are --include-package-data which will copy all non-DLLs and non-Python files of package names matching the pattern given. And --include-data-file takes source and relative target file paths and copies them. For onefile this is the only way to include files, for standalone mode they are mostly a convenience function.

  • Onefile: Added mode where the file is unpacked to a temporary folder before running instead of doing it to appdata.

  • Onefile: Added linux specific options --linux-onefile-icon to allow provision of an icon to use in onefile mode on Linux, so far this was only available as the hard coded path to a Python icon, which also didn't exist on all platforms.

  • UI: Major logging cleanup. Everything is now using our tracing classes and even error exits go through there and are therefore colored if possible.

  • Plugins: Make it easier to integrate commercial plugins, now only an environment variable needs to point to them.

  • UI: Enhanced option parsing gives notes. This complains about options that conflict or that are implied in others. Trying to catch more usage errors sooner.

  • Plugins: Ignore exceptions in buggy plugin code, only warn about them unless in debug mode, where they still crash Nuitka.

  • Scons: More complete scons report files, includes list values as well and more modes used.

  • Windows: The clcache is now included and no longer used from the system.

  • Output for clcache and ccache results got improved.

  • Enhanced support for clang, on Windows if present near a gcc.exe like it is the case for some winlibs downloads, it will be used. To use it provide --mingw64 --clang both. Without the first one, it will mean clangcl.exe which uses the MSVC compiler as a host.

Optimization

  • Some modules had very slow load times, e.g. if they used many list objects due to linear searches for memory deduplication of objects. We now have dictionaries of practically all constant objects loaded, making these more instant.

  • Use less memory at compile time due using __slots__ for all node types, finally figured out, how to achieve this with multiple inheritance.

  • Use hedley for compiler macros like unlikely as they know best how to do these.

  • Special case the merging of 2 branches avoiding generic code and being much faster.

  • Hard imports have better code generated, and are being optimized into for the few standard library modules and builtin modules we handle, they also now annotate the type shape to be module.

  • No longer annotate hard module import attribute lookups as control flow escapes. Not present attributes are changed into static raises. Trust for values is configured for a few values, and experimental.

  • Avoid preloaded packages for modules that have no side effects and are in the standard library, typically .pth files will use e.g. os but that's not needed to be preserved.

  • Use incbin for including binary data through inline assembly of the C compiler. This covers many more platforms than our previous linker option hacks, and the fallback to generated C code. In fact everything but Windows uses this now.

Organisational

  • Windows: For Scons we now require a Python 3.5 or higher to be installed to use it.

  • Windows: Removed support for gcc older than version 8. This specifically affects CondaCC and older MinGW64 installations. Since Nuitka can now download the MinGW64 10, there is no point in having these and they cause issues.

  • We took over the maintenance of clcache as Nuitka/clcache which is not yet ready for public consumption, but should become the new source of clache in the future.

  • Include an inline copy of clcache in Nuitka and use it on Windows for MSVC and ClangCL.

  • Removed compatibility older aliases of follow option, --recurse-* and require --follow-* options to be used instead.

  • For pylint checking, the tool now supports a --diff mode where only the changed files get checked. This is much faster and allows to do it more often before commit.

  • Check the versions of isort and black when doing the autoformat to avoid using outdated versions.

  • Handling missing pylint more gracefully when checking source code quality.

  • Make sure to use the codespell tool with Python3 and make sure to error exit when spelling problems were found, so we can use this in Github actions too.

  • Removed Travis config, we now only use Github actions.

  • Removed landscape config, it doesn't really exist anymore.

  • Bumped all PyPI dependnecies to their latest versions.

  • Recommend ccache on Debian, as we now consider the absence of ccache something to warn about.

  • Plugins: The DLLs asked for by plugins that are not found are no longer warned about.

  • Allow our checker and format tools to run on outside of tree code. We are using that for Nuitka/clcache.

  • Added support for Fedora 33 and openSUSE 15.3, as well as Ubuntu Groovy.

  • Windows: Check if Windows SDK is installed for MSVC and ClangCL.

  • Windows: Enhanced wording in case no compiler was found. No longer tell people how to manually install MinGW64, that is no longer necessary and pywin32 is not needed to detect MSVC, so it's not installed if not found.

  • Detect "embeddable Python" by missing include files, and reject it with proper error message.

  • Added onefile and standalone as a use case to the manual and put also the DLL and data files problems as typically issues.

Cleanups

  • Avoid decimal and string comparisons for Python versions checks, these were lazy and are going to break once 3.10 surfaces. In testing we now use tuples, in Nuitka core hexacimal values much like CPython itself does.

  • Stop using subnode child getters and setters, and instead only use subnode attributes. This was gradually changed so far, but in this release all remaining uses have migrated. This should also make the optimization stage go faster.

  • Change node constructors to not use a decorator to resolve conflicts with builtin names, rather handle these with manual call changes, the decorator only made it difficult to read and less performant.

  • Move safe string helpers to their own dedicated helper file, allowing for reuse in plugin code that doesn't want to use all of Nuitka C helpers.

  • Added utils code for inline copy imports, as we use that for quite a few things now.

  • Further restructured the Scons files to use more common code.

  • Plugins: The module name objects now reject many str specific APIs that ought to not be used, and the code got changed to use these instead, leading to cleaner and more correct usages.

  • Using named tuples to specify included data files and entry points.

  • Use pkgutil in plugins to scan for modules rather than listing directories.

Tests

  • New option to display executed commands during comparisons.

  • Added test suite for onefile testing.

Summary

This release has seen Python3.9 and Onefile both being completed. The later needs compression added on Windows, but that can be added in a coming release, for now it's fully functional.

The focus clearly has been on massive cleanups, some of which will affect compile time performance. There is relatively little new optimization otherwise.

The adoption of clcache enables a very fast caching, as it's now loaded directly into the Scons process, avoiding a separate process fork.

Generally a lot of polishing has been applied with many cleanups lowering the technical debt. It will be interesting to see where the hard module imports can lead us in terms of more optimization. Static optimization of the Python version comparisons and checks is needed to lower the amount of imports to be processed.

Important fixes are also included, e.g. the constants loading performance was too slow in some cases. The multiprocessing on Windows and numpy plugins were regressed and finally everything ought to be back to working fine.

Future work will have to aim at enhanced scalability. In some cases, Nuitka still takes too much time to compile if projects like Pandas include virtually everything installed as an option for it to use.

Nuitka Release 0.6.10

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release comes with many new features, e.g. onefile support, as well as many new optimization and bug fixes.

Bug Fixes

  • Fix, was memory leaking arguments of all complex call helper functions. Fixed in 0.6.9.6 already.

  • Plugins: Fix, the dill-compat code needs to follow API change. Fixed in 0.6.9.7 already.

  • Windows: Fixup for multiprocessing module and complex call helpers that could crash the program. Fixed in 0.6.9.7 already.

  • Fix, the frame caching could leak memory when using caching for functions and generators used in multiple threads.

  • Python3: Fix, importing an extension module below a compiled module was not possible in accelerated mode.

  • Python3: Fix, keyword arguments for open built-in were not fully compatible.

  • Fix, the scons python check should also not accept directories, otherwise strange misleading error will occur later.

  • Windows: When Python is installed through a symbolic link, MinGW64 and Scons were having issues, added a workaround to resolve it even on Python2.

  • Compatibility: Added support for co_freevars in code objects, e.g. newer matplotlib needs this.

  • Standalone: Add needed data files for gooey. Fixed in 0.6.9.4 already.

  • Scons: Fix, was not respecting --quiet option when running Scons. Fixed in 0.6.9.3 already.

  • Scons: Fix, wasn't automatically detecting Scons from promised paths. Fixed in 0.6.9.2 already.

  • Scons: Fix, the clcache output parsing wasn't robust enough. Fixed in 0.6.9.1 already.

  • Python3.8: Ignore all non-strings provided in doc-string fashion, they are not to be considered.

  • Fix, getattr, setattr and hasattr could not be used in finally clauses anymore. Fixed in 0.6.9.1 already.

  • Windows: For Python3 enhanced compatibility for Windows no console mode, they need a sys.stdin or else e.g. input will not be compatible and raise RuntimeError.

New Features

  • Added experimental support for Python 3.9, in such a way that the CPython3.8 test suite passes now, the 3.9 suite needs investigation still, so we might be missing new features.

  • Added experimental support for Onefile mode with --onefile that uses AppImage on Linux and our own bootstrap binary on Windows. Other platforms are not supported at this time. With this, the standalone folder is packed into a single binary. The Windows variant currently doesn't yet do any compression yet, but the Linux one does.

  • Windows: Added downloading of ccache.exe, esp. as the other sources so far recommended were not working properly after updates. This is taken from the official project and should be good.

  • Windows: Added downloading of matching MinGW64 C compiler, if no other was found, or that was has the wrong architecture, e.g. 32 bits where we need 64 bits.

  • Windows: Added ability to copy icon resources from an existing binary with new option --windows-icon-from-exe.

  • Windows: Added ability to provide multiple icon files for use with different desktop resolutions with new option --windows-icon-from-ico that got renamed to disambiguate from other icon options.

  • Windows: Added support for requesting UAC admin right with new option --windows-uac-admin.

  • Windows: Added support for requesting "uiaccess" rights with yet another new option --windows-uac-uiaccess.

  • Windows: Added ability to specify version info to the binary. New options --windows-company-name, --windows-product-name, --windows-file-version, --windows-product-version, and --windows-file-description have been added. Some of these have defaults.

  • Enhanced support for using the Win32 compiler of MinGW64, but it's not perfect yet and not recommended.

  • Windows: Added support for LTO mode for MSVC as well, this seems to allow more optimization.

  • Plugins: The numpy plugin now handles matplotlib3 config files correctly.

Optimization

  • Use less C variables in dictionary created, not one per key/value pair. This improved scalability of C compilation.

  • Use common code for module variable access, leading to more compact code and enhanced scalability of C compilation.

  • Use error exit during dictionary creation to release the dictionary, list, tuple, and set in case of an error occurring while they are still under construction. That avoids releases of it in error exists, reducing the generated code size by a lot. This improves scalability of C compilation for generating these.

  • Annotate no exception raise for local variables of classes with know dict shape, to avoid useless error exits.

  • Annotate no exception exit for staticmethod and classmethod as they do not check their arguments at all. This makes code generated for classes with these methods much more compact, mainly improving their scalability in C compilation.

  • In code generation, prefer bool over nuitka_bool which allows to annotate exception result, leading to more compact code. Also cleanup so that code generation always go through the C type objects, rather than doing cases locally, adding a C type for bool.

  • Use common code for C code handling const None return only, to cases where there is any immutable constant value returned, avoid code generation for this common case. Currently mutable constants are not handled, this may be added in the future.

  • Annotate no exception for exception type checks in handlers for Python2 and no exception if the value has exception type shape for Python3. The exception type shape was newly added. This avoids useless exception handlers in most cases, where the provided exception is just a built-in exception name.

  • Improve speed of often used compile time methods on nodes representing constant values, by making their implementation type specific to improve frontend compile time speed, we check e.g. mutable and hashable a lot.

  • Provide truth value for variable references, enhancing loop optimization and merge value tracing, to also decide this correctly for values only read, and then changed through attribute, e.g. append on lists. This allows many more static optimization.

  • Use staticmethod for methods in Nuitka nodes to achieve faster frontend compile times where possible.

  • Use dedicated helper code for calls with single argument, avoiding the need have a call site local C array of size one, just to pass a pointer to it.

  • Added handling of hash slot, to predict hashable keys for dictionary and sets.

  • Share more slot provision for built-in type shapes from mixin classes, to get them more universally provided, even for special types, where their consideration is unusual.

  • Trace "user provided" flag only for constants where it really matters, i.e. for containers and generally potentially large values, but not for every number or boolean value.

  • Added lowering of bytearray constant values to bytes value iteration, while handling constant values for this optimization with dedicated code for improved frontend compilation speed.

  • The dict built-in now annotates the dictionary type shape of its result.

  • The wrapping side-effects node now passes on the type shape of the wrapped value, allowing for optimization of these too.

  • Split slice nodes into variants with 1, 2 or 3 arguments, to avoid the overhead of determining which case we have, as well as to save a bit of memory, since these are more frequently used on Python3 for subscript operations. Also annotate their type shape, allowing more optimization.

  • Faster dictionary lookups, esp. in cases where errors occur, because we were manually recreating a KeyError that is already provided by the dict implementation. This should also be faster, as it avoids a CPython API call overhead on the DLL and they can provide a reference or not for the returned value, simplifying using code.

  • Faster dictionary containment checks, with our own dedicated helper, we can use code that won't create an exception when an item is not present at all.

  • Faster hash lookups with our own helper, separating cases where we want an exception for non-hashable values or not. These should also be faster to call.

  • Avoid acquiring thread state in exception handling that checks if a StopIteration occurred, to improved speed on Python3, where is involves locking, but this needs to be applied way more often.

  • Make sure checks to debug mode and full compatibility mode are done with the variables introduced, to avoid losing performance due to calls for Nuitka compile time enhancements. This was so far only done partially.

  • Split constant references into two base classes, only one of them tracking if the value was provided by the user. This saves compile time memory and avoids the overhead to check if sizes are exceeded in cases they cannot possibly be so.

  • The truth value of container creations is now statically known, because the empty container creation is no longer a possibility for these nodes, allowing more optimization for them.

  • Optimize the bool built-in with no arguments directory, allow to simplify the node for single argument form to avoid checks if an argument was given.

  • Added iteration handles for xranges, and make them faster to create by being tied to the node type, avoiding shared types, instead using the mixin approach. This is in preparation to using them for standard iterator tracing as well. So far they are only used for any and all decision.

  • Added detection if a iterator next can raise, using existing iterator checking which allows to remove needless checks and exception traces. Adding a code variant for calls to next that cannot fail, while tuning the code used for next and unpacking next, to use faster exception checking in the C code. This will speed up unpacking performance for some forms of unpacking from known sizes.

  • Make sure to use the fastest tuple API possible in all of Nuitka, many place e.g. used PyTuple_Size, and one was in a performance critical part, e.g. in code that used when compiled functions as called as a method.

  • Added optimized variant for _PyList_Extend for slightly faster unpacking code.

  • Added optimized variant for PyList_Append for faster list contractions code.

  • Avoid using RemoveFileSpec and instead provide our own code for that task, slightly reducing file size and avoiding to use the Shlapi link library.

Tests

  • Made reflected test use common cleanup of test folder, which is more robust against Windows locking issues.

  • Only output changed CPython output after the forced update of cached value was done, avoiding duplicate or outdated outputs.

  • Avoid complaining about exceptions for in-place operations in case they are lowered to non-inplace operations and then raise unsupported, not worth the effort to retain original operator.

  • Added generated test for subscript operations, also expanding coverage in generated tests by making sure, conditional paths are both taken by varying the cond value.

  • Use our own code helper to check if an object has an attribute, which is faster, because it avoids creating exceptions in the first place, instead of removing them afterwards.

Cleanups

  • Make sure that code generation always go through the C type objects rather than local elif casing of the type. This required cleaning up many of the methods and making code more abstract.

  • Added base class for C types without reference counting, so they can share the code that ignores their handling.

  • Remove getConstant for constant value nodes, use the more general getCompileTimeConstant instead, and provide quick methods that test for empty tuple or dict, to use for checking concrete values, e.g. with call operations.

  • Unified container creation into always using a factory function, to be sure that existing container creations are not empty.

  • Stop using @calledWithBuiltinArgumentNamesDecorator where possible, and instead make explicit wrapping or use correct names. This was used to allow e.g. an argument named list to be passed from built-in optimization, but that can be done in a cleaner fashion. Also aligned no attributes and the argument names, there was inconsistency there.

  • Name mangling was done differently for attribute names and normal names and with non-shared code, and later than necessary, removing this as a step from variable closure taking after initial tree build.

  • As part of the icon changes, now handled in Python code, we stop using the rc binary and handle all resources ourselves, allowing to remove that code from the Scons side of things.

  • Moved file comparison code of standalone mode into file utils function for use in plugins as well.

  • Unified how path concatenation is done in Nuitka helper code, there were more or less complete variants, this is making sure, the most capable form is used in all cases.

  • Massive cleanup to our scons file, by moving out util code that only scons uses, hacks we apply to speed up scons, and more to separate modules with dedicated interfaces.

  • When using enumerate we now provide start value of 1 where it is appropriate, e.g. when counting source code lines, rather than adding count+1 on every usage, making code more readable.

Organisational

  • Do not recommend Anaconda on Windows anymore, it seems barely possible to get anything installed on it with a fresh download, due to the resolver literally working for days without finishing, and then reporting conflicts, it would only we usable when starting with Miniconda, but that seems less interesting to users, also gcc 5.2 is way too old these days.

  • The commit hook should be reinstalled, since it got improved and adapted for newer git versions.

  • Added link to donations to funding document, following a Github standard.

  • Bumped requirements for development to the latest versions, esp. newer isort.

  • Added a rough description of tests to do to add a new CPython test suite, to allow others to take this task in the future.

  • Updated the git hook so that Windows and newest git works.

  • Make it more clear in the documentation that Microsoft Appstore Python is not supported.

Summary

This is the big release in terms of scalability. The optimization in this release mostly focused on getting things that cause increased compile times sorted out. A very important fix avoids loop optimization to leak into global passes of all modules unnecessarily, but just as important, generated code now is much better for the C compiler to consume in observed problematic cases.

More optimization changes are geared towards reducing Nuitka frontend compile time, which could also be a lot in some cases, ending up specializing more constant nodes and how they expose themselves to optimization.

Other optimization came from supporting Python 3.9 and things come across during the implementation of that feature, e.g. to be able to make differences with unpacking error messages, we provide more code to handle it ourselves, and to manually optimize how to interact with e.g. list objects.

For Windows, the automatic download of ccache and a matching MinGW64 if none was found, is a new step, that should lower the barrier of entry for people who have no clue what a C compiler is. More changes are bound to come in this field with future releases, e.g. making a minimum version requirement for gcc on Windows that excludes unfit C compilers.

All in all, this release should be taken as a major cleanup, resolving many technical debts of Nuitka and preparing more optimization to come.

Nuitka Release 0.6.9

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This releases contains important bug fixes for regressions of the 0.6.8 series which had relatively many problems. Not all of these could be addressed as hotfixes, and other issues were even very involved, causing many changes to be necessary.

There are also many general improvements and performance work for tracing and loops, but the full potential of this will not be unlocked with this release yet.

Bug Fixes

  • Fix, loop optimization sometimes didn't determinate, effectively making Nuitka run forever, with no indication why. This has been fixed and a mechanism to give up after too many attempts has been added.

  • Fix, closure taking object allowed a brief period where the garbage collector was exposed to uninitialized objects. Fixed in 0.6.8.1 already.

  • Python3.6+: Fix corruption for exceptions thrown into asyncgen. Fixed in 0.6.8.1 already.

  • Fix, deleting variables detected as C type bool could raise an UnboundLocalError that was wrong. Fixed in 0.6.8.1 already.

  • Python3.8.3+: Fix, future annotations parsing was using hard coded values that were changed in CPython, leading to errors.

  • Windows: Avoid encoding issues for Python3 on more systems, by going from wide characters to unicode strings more directly, avoiding an encoding as UTF8 in the middle. Fixed in 0.6.8.2 already.

  • Windows: Do not crash when warning about uninstalled MSVC using Python3. This is a Scons bug that we fixed. Fixed in 0.6.8.3 already.

  • Standalone: The output of dependency walker should be considered as "latin1" rather than UTF8. Fixed in 0.6.8.3 already.

  • Standalone: Added missing hidden dependencies for flask. Fixed in 0.6.8.1 already.

  • Standalone: Fixed win32com.client on Windows. Fixed in 0.6.8.1 already.

  • Standalone: Use pkgutil to scan encoding modules, properly ignoring the same files as Python does in case of garbage files being there. Fixed in 0.6.8.2 already.

  • Plugins: Enabling a plugin after the filename to compile was given, didn't allow for arguments to the passed, causing problems. Fixed in 0.6.8.3 already.

  • Standalone: The certifi data file is now supported for all modules using it and not only some.

  • Standalone: The bytecode for the standard library had filenames pointing to the original installation attached. While these were not used, but replaced at runtime, they increased the size of the binary, and leaked information.

  • Standalone: The path of sys.executable was not None, but pointing to the original executable, which could also point to some temporary virtualenv directory and therefore not exist, also it was leaking information about the original install.

  • Windows: With the MSVC compiler, elimination of duplicate strings was not active, causing even unused strings to be present in the binary, some of which contained file paths of the Nuitka installation.

  • Standalone: Added support for pyglet.

  • Plugins: The command line handling for Pmw plugin was using wrong defaults, making it include more code than necessary, and to crash if it was not there.

New Features

  • Windows: Added support for using Python 2.7 through a symlink too. This was already working for Python3, but a scons problem prevented this from working.

  • Caching of compiled C files is now checked with ccache and clcache, and added automatically where possible, plus a report of the success is made. This can accelerate the re-compile very much, even if you have to go through Nuitka compilation itself, which is not (yet) cached.

  • Added new --quiet option that will disable informational traces that are going to become more.

  • The Clang from MSVC installation is now picked up for both 32 and 64 bits and follows the new location in latest Visual Studio 2019.

  • Windows: The ccache from Anaconda is now supported as well as the one from msys64.

Optimization

  • The value tracing has become more correct with loops and in general less often inhibits optimization. Escaping of value traces is now a separate trace state allowing for more appropriate handling of actual unknowns.

  • Memory used for value tracing has been lowered by removing unnecessary states for traces, that we don't use anymore.

  • Windows: Prevent scons from scanning for MSVC when asked to use MinGW64. This avoids a performance loss doing something that will then end up being unused.

  • Windows: Use function level linking with MSVC, this will allow for smaller binaries to be created, that don't have to include unused helper functions.

Cleanups

  • The scons file now uses Nuitka utils functions and is itself split up into several modules for enhanced readability.

  • Plugin interfaces for providing extra entry points have been cleaned up and now named tuples are used. Backward compatibility is maintained though.

Organisational

  • The use of the logging module was replaced with more of our custom tracing and we now have the ability to write the optimization log to a separate file.

  • Old style plugin options are now detected and reported as a usage error rather than unknown plugin.

  • Changed submodules to use git over https, so as to not require ssh which requires a key registered and causes problems with firewalls too.

  • More correct Debian copyright file, made formatting of emails in source code consistent.

  • Added repository for Ubuntu focal.

Summary

The main focus of this release has been bug fixes with only a little performance work due to the large amount of regressions and other findings from the last release.

The new constants loading for removes a major scalability problem. The checked and now consistently possible use of ccache and clcache allows for much quicker recompilation. Nuitka itself can still be slow in some cases, but should have seen some improvements too. Scalability will have to remain a focus for the next releases too.

The other focus, was to make the binaries contain no original path location, which is interesting for standalone mode. Nuitka should be very good in this area now.

For optimization, the new loop code is again better. But it was also very time consuming, to redo it, yet again. This has prevented other optimization to be added.

And then for correctness, the locals scope work, while very invasive, was necessary, to handle the usage of locals inside of contractions, but also will be instrumental for function inlining to become generally available.

So, ultimately, this release is a necessary intermediate step. Upcoming releases will be able to focus more clearly on run time performance again as well as on scalability for generated C code.

Nuitka Release 0.6.8

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This releases contains important general improvements and performance improvements and enhanced optimization as well as many bug fixes that enhance the Python 3.8 compatibility.

Bug Fixes

  • Python3.5+: Fix, coroutines and asyncgen could continue iteration of awaited functions, even after their return, leading to wrong behaviour.

  • Python3.5+: Fix, absolute imports of names might also refer to modules and need to be handled for module loading as well.

  • Fix, the fromlist of imports could loose references, potentially leading to corruption of contained strings.

  • Python3.8: Fix, positional only arguments were not enforced to actually be that way.

  • Python3.8: Fix, complex calls with star arguments that yielded the same value twice, were not yet caught.

  • Python3.8: Fix, evaluation order for nested dictionary contractions was not followed yet.

  • Windows: Use short paths, these work much better to load extension modules and TCL parts of TkInter cannot handle unicode paths at all. This makes Nuitka work in locations, where normal Python cannot.

  • Windows: Fixup dependency walker in unicode input directories.

  • Standalone: Use frozen module loader only at libpython initialisation and switch to built-in bytecode loader that is more compatible afterwards, increasing compatibility.

  • Standalone: Fix for pydanctic support.

  • Standalone: Added missing hidden dependency of uvicorn.

  • Fix, the parser for .pyi files couldn't handle multiline imports.

  • Windows: Derive linker arch of Python from running binary, since it can happen that the Python binary is actually a script.

  • Fixup static linking with libpython.a that contains main.o by making our colliding symbols for Py_GetArgcArgv weak.

  • Python3.7: Fix misdetection as asyncgen for a normal generator, if the iterated value is async.

  • Distutils: Fix build_nuitka for modules under nested namespaces.

  • OpenBSD: Follow usage of clang and other corrections to make accelerated mode work.

  • macOS: Fixup for standalone mode library scan.

  • Fix, the logging of --show-modules was broken.

  • Windows: Enable /bigobj mode for MSVC for large compilations to work.

  • Windows: Fixup crash in warning with pefile dependency manager.

  • Windows: Fixup win32com standalone detection of other Python version win32com is in system PATH.

  • Fix, the python flag for static hashes didn't have the intended effect.

  • Fix, generators may be resurrected in the cause of their destruction, and then must not be released.

  • Fix, method objects didn't implement the methods __reduce__ and __reduce_ex__ necessary for pickling them.

  • Windows: Fix, using a Python installation through a symlink was not working.

  • Windows: Fix, icon paths that were relative were not working anymore.

  • Python3.8: Detect duplicate keywords yielded from star arguments.

  • Fix, methods could not be pickled.

  • Fix, generators, coroutines and asyncgen might be resurrected during their release, allow for that.

  • Fix, frames need to traverse their attached locals to be released in some cases.

New Features

  • Plugin command line handling now allows for proper optparse options to be used, doing away with special parameter code for plugins. The arguments now also become automatically passed to the instantiations of plugins.

    Loading and creation of plugins are now two separate phases. They are loaded when they appear on the command line and can add options in their own group, even required ones, but also with default values.

  • Started using logging with name-spaces. Applying logging per plugin to make it easier to recognize which plugin said what. Warnings are now colored in red.

  • Python3.5+: Added support for two step module loading, making Nuitka loading even more compatible.

  • Enhanced import tracing to work on standalone binaries in a useful manner, allow to compare with normal binaries.

  • Fix, the setattr built-in was leaking a reference to the None value.

Optimization

  • Proper loop SSA capable of detecting shapes with an incremental initial phase and a final result of alternatives for variables written in the loop. This detects shapes of manual integer incrementing loops correctly now, it doesn't see through iterators yet, but this will come too.

  • Added type shapes for all operations and all important built-in types to allow more compile time optimization and better target type selection.

  • Target type code generation was expanded from manual usage with conditions to all operations allowing to get at bool target values more directly.

  • For in-place operations, there is the infrastructure to generate them for improved performance, but so far it's only used for Python2 int, and not for the many types normal operations are supported.

  • Force usage of C boolean type for all indicator variables from the re-formulation. In some cases, we are not yet there with detections, and this gives instant benefit.

  • Complex constants didn't annotate their type shape, preventing compile time optimization for them.

  • Python3.8: Also support vectorcall for compiled method objects. These are rarely used in new Python, but can make a difference.

  • Remove loops that have only a final break. This happens in static optimization in some cases, and allows more optimization to be done.

  • Avoid using a preparing a constant tuple value for calls with only constant arguments.

  • Avoid using PyErr_Format where it's not necessary by adding specialized helpers for common cases.

  • Detect del statements that will raise an exception and replace with that.

  • Exception matching is boolean shape, allowing for faster code generation.

  • Disable recursion checks outside of full compat mode.

  • Avoid large blocks for conditional statements that only need to enclose the condition evaluation.

  • Added shortcuts for interactions between compiled generator variants, to avoid calls to their C methods with argument passing, etc.

Organisational

  • Updated developer manual with changes that happened, remvoing the obsolete language choice section.

  • Added 3.8 support mentions is even more places.

  • The mailing list has been deleted. We now prefer Gitter chat and Github issues for discussions.

  • Visual Code recommended extensions are now defined as such in the project configuration and you will be prompted to install them.

  • Visual Code environents for Py38 and Py27 were added for easier switch.

  • Catch usage of Python from the Microsoft App Store, it is not supported and seems to limit access to the Python installation for security reasons that make support impossible.

  • Make it clear that --full-compat should not be used in help output.

  • Added instructions for MSVC runtimes and standalone compilation to support Windows 7.

  • More complete listing of copyright holders for Debian.

  • Updated to newer black and PyLint.

  • Enhanced gcc version check, properly works with gcc 10 and higher.

Tests

  • Pylint cleanups for some of the tests.

  • Added test for loading of user plugins.

  • Removed useless outputs for search mode skipping non-matches.

Cleanups

  • Limit command line handling for multiprocessing module to when the plugin is actually used, avoiding useless code of Windows binaries.

  • Pylint cleanup also foreign code like oset and odict.

  • In preparation of deprecating the alternative, --plugin-enable has become the only form used in documentation and tests.

  • Avoid numeric pylint symbols more often.

  • Distutils: Cleanup module name for distutils commands, these are not actually enforced by distutils, but very ugly in our coding conventions.

  • The "cannot get here" code to mark unreachable code has been improved and no longer needs an identifier passed, but uses the standard C mechanism for that.

  • Removed accessors for lookup sources from nodes, allowing for faster usage and making sure, lookups are only done where needed.

Summary

This release is huge in terms of bugs fixed, but also extremely important, because the new loop SSA and type tracing, allows for many more specialized code usages. We now can trace the type for some loops to be specifically an integer or long value only, and will become able to generate code that avoids using Python objects, in these cases.

Once that happens, the performance will make a big jump. Future releases will have to consolidate the current state, but it is expected that at least an experimental addition of C type float or C long can be added, add to that iterator type shape and value analsis, and an actual jump in performance can be expected.

Nuitka Release 0.6.7

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains bug fixes and improvements to the packaging, for the RPM side as well as for Debian, to cover Python3 only systems as they are now becoming more common.

Bug Fixes

  • Compatibility: The value of __module__ for extension modules was not dependent into which package the module was loaded, it now is.

  • Anaconda: Enhanced detection of Anaconda for Python 3.6 and higher.

  • CentOS6: Detect gcc version to allow saving on macro memory usage, very old gcc didn't have that.

  • Include Python3 for all Fedora versions where it works as well as for openSUSE versions 15 and higher.

  • Windows: Using short path names to interact with Scons avoids problems with unicode paths in all cases.

  • macOS: The usage of install_name_tool could sometimes fail due to length limits, we now increase it at link time.

  • macOS: Do not link against libpython for module mode. This prevented extension modules from actually being usable.

  • Python3.6: Follow coroutine fixes in our asyncgen implementation as well.

  • Fix, our version number handling could overflow with minor versions past 10, so we limited it for now.

New Features

  • Added support for Python 3.8, the experimental was already there and pretty good, but now added the last obscure features too.

  • Plugins can now provide C code to be included in the compilation.

  • Distutils: Added targets build_nuitka and install_nuitka to complement bdist_nuitka, so we support software other than wheels, e.g. RPM packaging that compiles with Nuitka.

  • Added support for lldb the Clang debugger with the --debugger mode.

Optimization

  • Make the file prefix map actually work for gcc and clang, and compile files inside the build folder, unless we are running in debugger mode, so we use ccache caching across different compilations for at least the static parts.

  • Avoid compilation of __frozen.c in accelerated mode, it's not used.

  • Prefer using the inline copy of scons over systems scons. The later will only be slower. Use the fallback to external scons only from the Debian packages, since there we consider it forbidden to include software as a duplicate.

Organisational

  • Added recommended plugins for Visual Code, replacing the list in the Developer Manual.

  • Added repository for Fedora 30 for download.

  • Added repository for CentOS 8 for download.

  • Updated inline copy of Scons used for Python3 to 3.1.2, which is said to be faster for large compilations.

  • Removed Eclipse setup from the manual, it's only infererior at this point and we do not use it ourselves.

  • Debian: Stop recommending PyQt5 in the package, we no longer use it for built-in GUI that was removed.

  • Debian: Bumped the standards version and modernized the packaging, solving a few warnings during the build.

Cleanups

  • Scons: Avoid to add Unix only include paths on Windows.

  • Scons: Have the static source code in a dedicated folder for clarity.

Tests

  • Added tests to Github Actions, for the supported Python versions for all of Linux, macOS and Windows, covering the later publicly for the first time. We use Anaconda on macOS for the tests now, rather than Homebrew.

  • Enable IO encoding to make sure we use UTF8 for more test suites that actually need it in case of problems.

  • Comparing module outputs now handles segfaults by running in the debugger too.

Summary

This release adds full support for Python 3.8 finally, which took us a while, and it cleans up a lot on the packaging side. There aren't that many important bug fixes, but it's still nice to this cleaned up.

We have important actual optimization in the pipeline that will apply specialization to target types and for comparison operations. We expect to see actual performance improvements in the next release again.