Nuitka Release 0.6.10

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release comes with many new features, e.g. onefile support, as well as many new optimization and bug fixes.

Bug Fixes

  • Fix, was memory leaking arguments of all complex call helper functions. Fixed in 0.6.9.6 already.

  • Plugins: Fix, the dill-compat code needs to follow API change. Fixed in 0.6.9.7 already.

  • Windows: Fixup for multiprocessing module and complex call helpers that could crash the program. Fixed in 0.6.9.7 already.

  • Fix, the frame caching could leak memory when using caching for functions and generators used in multiple threads.

  • Python3: Fix, importing an extension module below a compiled module was not possible in accelerated mode.

  • Python3: Fix, keyword arguments for open built-in were not fully compatible.

  • Fix, the scons python check should also not accept directories, otherwise strange misleading error will occur later.

  • Windows: When Python is installed through a symbolic link, MinGW64 and Scons were having issues, added a workaround to resolve it even on Python2.

  • Compatibility: Added support for co_freevars in code objects, e.g. newer matplotlib needs this.

  • Standalone: Add needed data files for gooey. Fixed in 0.6.9.4 already.

  • Scons: Fix, was not respecting --quiet option when running Scons. Fixed in 0.6.9.3 already.

  • Scons: Fix, wasn't automatically detecting Scons from promised paths. Fixed in 0.6.9.2 already.

  • Scons: Fix, the clcache output parsing wasn't robust enough. Fixed in 0.6.9.1 already.

  • Python3.8: Ignore all non-strings provided in doc-string fashion, they are not to be considered.

  • Fix, getattr, setattr and hasattr could not be used in finally clauses anymore. Fixed in 0.6.9.1 already.

  • Windows: For Python3 enhanced compatibility for Windows no console mode, they need a sys.stdin or else e.g. input will not be compatible and raise RuntimeError.

New Features

  • Added experimental support for Python 3.9, in such a way that the CPython3.8 test suite passes now, the 3.9 suite needs investigation still, so we might be missing new features.

  • Added experimental support for Onefile mode with --onefile that uses AppImage on Linux and our own bootstrap binary on Windows. Other platforms are not supported at this time. With this, the standalone folder is packed into a single binary. The Windows variant currently doesn't yet do any compression yet, but the Linux one does.

  • Windows: Added downloading of ccache.exe, esp. as the other sources so far recommended were not working properly after updates. This is taken from the official project and should be good.

  • Windows: Added downloading of matching MinGW64 C compiler, if no other was found, or that was has the wrong architecture, e.g. 32 bits where we need 64 bits.

  • Windows: Added ability to copy icon resources from an existing binary with new option --windows-icon-from-exe.

  • Windows: Added ability to provide multiple icon files for use with different desktop resolutions with new option --windows-icon-from-ico that got renamed to disambiguate from other icon options.

  • Windows: Added support for requesting UAC admin right with new option --windows-uac-admin.

  • Windows: Added support for requesting "uiaccess" rights with yet another new option --windows-uac-uiaccess.

  • Windows: Added ability to specify version info to the binary. New options --windows-company-name, --windows-product-name, --windows-file-version, --windows-product-version, and --windows-file-description have been added. Some of these have defaults.

  • Enhanced support for using the Win32 compiler of MinGW64, but it's not perfect yet and not recommended.

  • Windows: Added support for LTO mode for MSVC as well, this seems to allow more optimization.

  • Plugins: The numpy plugin now handles matplotlib3 config files correctly.

Optimization

  • Use less C variables in dictionary created, not one per key/value pair. This improved scalability of C compilation.

  • Use common code for module variable access, leading to more compact code and enhanced scalability of C compilation.

  • Use error exit during dictionary creation to release the dictionary, list, tuple, and set in case of an error occurring while they are still under construction. That avoids releases of it in error exists, reducing the generated code size by a lot. This improves scalability of C compilation for generating these.

  • Annotate no exception raise for local variables of classes with know dict shape, to avoid useless error exits.

  • Annotate no exception exit for staticmethod and classmethod as they do not check their arguments at all. This makes code generated for classes with these methods much more compact, mainly improving their scalability in C compilation.

  • In code generation, prefer bool over nuitka_bool which allows to annotate exception result, leading to more compact code. Also cleanup so that code generation always go through the C type objects, rather than doing cases locally, adding a C type for bool.

  • Use common code for C code handling const None return only, to cases where there is any immutable constant value returned, avoid code generation for this common case. Currently mutable constants are not handled, this may be added in the future.

  • Annotate no exception for exception type checks in handlers for Python2 and no exception if the value has exception type shape for Python3. The exception type shape was newly added. This avoids useless exception handlers in most cases, where the provided exception is just a built-in exception name.

  • Improve speed of often used compile time methods on nodes representing constant values, by making their implementation type specific to improve frontend compile time speed, we check e.g. mutable and hashable a lot.

  • Provide truth value for variable references, enhancing loop optimization and merge value tracing, to also decide this correctly for values only read, and then changed through attribute, e.g. append on lists. This allows many more static optimization.

  • Use staticmethod for methods in Nuitka nodes to achieve faster frontend compile times where possible.

  • Use dedicated helper code for calls with single argument, avoiding the need have a call site local C array of size one, just to pass a pointer to it.

  • Added handling of hash slot, to predict hashable keys for dictionary and sets.

  • Share more slot provision for built-in type shapes from mixin classes, to get them more universally provided, even for special types, where their consideration is unusual.

  • Trace "user provided" flag only for constants where it really matters, i.e. for containers and generally potentially large values, but not for every number or boolean value.

  • Added lowering of bytearray constant values to bytes value iteration, while handling constant values for this optimization with dedicated code for improved frontend compilation speed.

  • The dict built-in now annotates the dictionary type shape of its result.

  • The wrapping side-effects node now passes on the type shape of the wrapped value, allowing for optimization of these too.

  • Split slice nodes into variants with 1, 2 or 3 arguments, to avoid the overhead of determining which case we have, as well as to save a bit of memory, since these are more frequently used on Python3 for subscript operations. Also annotate their type shape, allowing more optimization.

  • Faster dictionary lookups, esp. in cases where errors occur, because we were manually recreating a KeyError that is already provided by the dict implementation. This should also be faster, as it avoids a CPython API call overhead on the DLL and they can provide a reference or not for the returned value, simplifying using code.

  • Faster dictionary containment checks, with our own dedicated helper, we can use code that won't create an exception when an item is not present at all.

  • Faster hash lookups with our own helper, separating cases where we want an exception for non-hashable values or not. These should also be faster to call.

  • Avoid acquiring thread state in exception handling that checks if a StopIteration occurred, to improved speed on Python3, where is involves locking, but this needs to be applied way more often.

  • Make sure checks to debug mode and full compatibility mode are done with the variables introduced, to avoid losing performance due to calls for Nuitka compile time enhancements. This was so far only done partially.

  • Split constant references into two base classes, only one of them tracking if the value was provided by the user. This saves compile time memory and avoids the overhead to check if sizes are exceeded in cases they cannot possibly be so.

  • The truth value of container creations is now statically known, because the empty container creation is no longer a possibility for these nodes, allowing more optimization for them.

  • Optimize the bool built-in with no arguments directory, allow to simplify the node for single argument form to avoid checks if an argument was given.

  • Added iteration handles for xranges, and make them faster to create by being tied to the node type, avoiding shared types, instead using the mixin approach. This is in preparation to using them for standard iterator tracing as well. So far they are only used for any and all decision.

  • Added detection if a iterator next can raise, using existing iterator checking which allows to remove needless checks and exception traces. Adding a code variant for calls to next that cannot fail, while tuning the code used for next and unpacking next, to use faster exception checking in the C code. This will speed up unpacking performance for some forms of unpacking from known sizes.

  • Make sure to use the fastest tuple API possible in all of Nuitka, many place e.g. used PyTuple_Size, and one was in a performance critical part, e.g. in code that used when compiled functions as called as a method.

  • Added optimized variant for _PyList_Extend for slightly faster unpacking code.

  • Added optimized variant for PyList_Append for faster list contractions code.

  • Avoid using RemoveFileSpec and instead provide our own code for that task, slightly reducing file size and avoiding to use the Shlapi link library.

Tests

  • Made reflected test use common cleanup of test folder, which is more robust against Windows locking issues.

  • Only output changed CPython output after the forced update of cached value was done, avoiding duplicate or outdated outputs.

  • Avoid complaining about exceptions for in-place operations in case they are lowered to non-inplace operations and then raise unsupported, not worth the effort to retain original operator.

  • Added generated test for subscript operations, also expanding coverage in generated tests by making sure, conditional paths are both taken by varying the cond value.

  • Use our own code helper to check if an object has an attribute, which is faster, because it avoids creating exceptions in the first place, instead of removing them afterwards.

Cleanups

  • Make sure that code generation always go through the C type objects rather than local elif casing of the type. This required cleaning up many of the methods and making code more abstract.

  • Added base class for C types without reference counting, so they can share the code that ignores their handling.

  • Remove getConstant for constant value nodes, use the more general getCompileTimeConstant instead, and provide quick methods that test for empty tuple or dict, to use for checking concrete values, e.g. with call operations.

  • Unified container creation into always using a factory function, to be sure that existing container creations are not empty.

  • Stop using @calledWithBuiltinArgumentNamesDecorator where possible, and instead make explicit wrapping or use correct names. This was used to allow e.g. an argument named list to be passed from built-in optimization, but that can be done in a cleaner fashion. Also aligned no attributes and the argument names, there was inconsistency there.

  • Name mangling was done differently for attribute names and normal names and with non-shared code, and later than necessary, removing this as a step from variable closure taking after initial tree build.

  • As part of the icon changes, now handled in Python code, we stop using the rc binary and handle all resources ourselves, allowing to remove that code from the Scons side of things.

  • Moved file comparison code of standalone mode into file utils function for use in plugins as well.

  • Unified how path concatenation is done in Nuitka helper code, there were more or less complete variants, this is making sure, the most capable form is used in all cases.

  • Massive cleanup to our scons file, by moving out util code that only scons uses, hacks we apply to speed up scons, and more to separate modules with dedicated interfaces.

  • When using enumerate we now provide start value of 1 where it is appropriate, e.g. when counting source code lines, rather than adding count+1 on every usage, making code more readable.

Organisational

  • Do not recommend Anaconda on Windows anymore, it seems barely possible to get anything installed on it with a fresh download, due to the resolver literally working for days without finishing, and then reporting conflicts, it would only we usable when starting with Miniconda, but that seems less interesting to users, also gcc 5.2 is way too old these days.

  • The commit hook should be reinstalled, since it got improved and adapted for newer git versions.

  • Added link to donations to funding document, following a Github standard.

  • Bumped requirements for development to the latest versions, esp. newer isort.

  • Added a rough description of tests to do to add a new CPython test suite, to allow others to take this task in the future.

  • Updated the git hook so that Windows and newest git works.

  • Make it more clear in the documentation that Microsoft Appstore Python is not supported.

Summary

This is the big release in terms of scalability. The optimization in this release mostly focused on getting things that cause increased compile times sorted out. A very important fix avoids loop optimization to leak into global passes of all modules unnecessarily, but just as important, generated code now is much better for the C compiler to consume in observed problematic cases.

More optimization changes are geared towards reducing Nuitka frontend compile time, which could also be a lot in some cases, ending up specializing more constant nodes and how they expose themselves to optimization.

Other optimization came from supporting Python 3.9 and things come across during the implementation of that feature, e.g. to be able to make differences with unpacking error messages, we provide more code to handle it ourselves, and to manually optimize how to interact with e.g. list objects.

For Windows, the automatic download of ccache and a matching MinGW64 if none was found, is a new step, that should lower the barrier of entry for people who have no clue what a C compiler is. More changes are bound to come in this field with future releases, e.g. making a minimum version requirement for gcc on Windows that excludes unfit C compilers.

All in all, this release should be taken as a major cleanup, resolving many technical debts of Nuitka and preparing more optimization to come.

Nuitka Release 0.6.9

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This releases contains important bug fixes for regressions of the 0.6.8 series which had relatively many problems. Not all of these could be addressed as hotfixes, and other issues were even very involved, causing many changes to be necessary.

There are also many general improvements and performance work for tracing and loops, but the full potential of this will not be unlocked with this release yet.

Bug Fixes

  • Fix, loop optimization sometimes didn't determinate, effectively making Nuitka run forever, with no indication why. This has been fixed and a mechanism to give up after too many attempts has been added.

  • Fix, closure taking object allowed a brief period where the garbage collector was exposed to uninitialized objects. Fixed in 0.6.8.1 already.

  • Python3.6+: Fix corruption for exceptions thrown into asyncgen. Fixed in 0.6.8.1 already.

  • Fix, deleting variables detected as C type bool could raise an UnboundLocalError that was wrong. Fixed in 0.6.8.1 already.

  • Python3.8.3+: Fix, future annotations parsing was using hard coded values that were changed in CPython, leading to errors.

  • Windows: Avoid encoding issues for Python3 on more systems, by going from wide characters to unicode strings more directly, avoiding an encoding as UTF8 in the middle. Fixed in 0.6.8.2 already.

  • Windows: Do not crash when warning about uninstalled MSVC using Python3. This is a Scons bug that we fixed. Fixed in 0.6.8.3 already.

  • Standalone: The output of dependency walker should be considered as "latin1" rather than UTF8. Fixed in 0.6.8.3 already.

  • Standalone: Added missing hidden dependencies for flask. Fixed in 0.6.8.1 already.

  • Standalone: Fixed win32com.client on Windows. Fixed in 0.6.8.1 already.

  • Standalone: Use pkgutil to scan encoding modules, properly ignoring the same files as Python does in case of garbage files being there. Fixed in 0.6.8.2 already.

  • Plugins: Enabling a plugin after the filename to compile was given, didn't allow for arguments to the passed, causing problems. Fixed in 0.6.8.3 already.

  • Standalone: The certifi data file is now supported for all modules using it and not only some.

  • Standalone: The bytecode for the standard library had filenames pointing to the original installation attached. While these were not used, but replaced at runtime, they increased the size of the binary, and leaked information.

  • Standalone: The path of sys.executable was not None, but pointing to the original executable, which could also point to some temporary virtualenv directory and therefore not exist, also it was leaking information about the original install.

  • Windows: With the MSVC compiler, elimination of duplicate strings was not active, causing even unused strings to be present in the binary, some of which contained file paths of the Nuitka installation.

  • Standalone: Added support for pyglet.

  • Plugins: The command line handling for Pmw plugin was using wrong defaults, making it include more code than necessary, and to crash if it was not there.

New Features

  • Windows: Added support for using Python 2.7 through a symlink too. This was already working for Python3, but a scons problem prevented this from working.

  • Caching of compiled C files is now checked with ccache and clcache, and added automatically where possible, plus a report of the success is made. This can accelerate the re-compile very much, even if you have to go through Nuitka compilation itself, which is not (yet) cached.

  • Added new --quiet option that will disable informational traces that are going to become more.

  • The Clang from MSVC installation is now picked up for both 32 and 64 bits and follows the new location in latest Visual Studio 2019.

  • Windows: The ccache from Anaconda is now supported as well as the one from msys64.

Optimization

  • The value tracing has become more correct with loops and in general less often inhibits optimization. Escaping of value traces is now a separate trace state allowing for more appropriate handling of actual unknowns.

  • Memory used for value tracing has been lowered by removing unnecessary states for traces, that we don't use anymore.

  • Windows: Prevent scons from scanning for MSVC when asked to use MinGW64. This avoids a performance loss doing something that will then end up being unused.

  • Windows: Use function level linking with MSVC, this will allow for smaller binaries to be created, that don't have to include unused helper functions.

Cleanups

  • The scons file now uses Nuitka utils functions and is itself split up into several modules for enhanced readability.

  • Plugin interfaces for providing extra entry points have been cleaned up and now named tuples are used. Backward compatibility is maintained though.

Organisational

  • The use of the logging module was replaced with more of our custom tracing and we now have the ability to write the optimization log to a separate file.

  • Old style plugin options are now detected and reported as a usage error rather than unknown plugin.

  • Changed submodules to use git over https, so as to not require ssh which requires a key registered and causes problems with firewalls too.

  • More correct Debian copyright file, made formatting of emails in source code consistent.

  • Added repository for Ubuntu focal.

Summary

The main focus of this release has been bug fixes with only a little performance work due to the large amount of regressions and other findings from the last release.

The new constants loading for removes a major scalability problem. The checked and now consistently possible use of ccache and clcache allows for much quicker recompilation. Nuitka itself can still be slow in some cases, but should have seen some improvements too. Scalability will have to remain a focus for the next releases too.

The other focus, was to make the binaries contain no original path location, which is interesting for standalone mode. Nuitka should be very good in this area now.

For optimization, the new loop code is again better. But it was also very time consuming, to redo it, yet again. This has prevented other optimization to be added.

And then for correctness, the locals scope work, while very invasive, was necessary, to handle the usage of locals inside of contractions, but also will be instrumental for function inlining to become generally available.

So, ultimately, this release is a necessary intermediate step. Upcoming releases will be able to focus more clearly on run time performance again as well as on scalability for generated C code.

Nuitka Release 0.6.8

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This releases contains important general improvements and performance improvements and enhanced optimization as well as many bug fixes that enhance the Python 3.8 compatibility.

Bug Fixes

  • Python3.5+: Fix, coroutines and asyncgen could continue iteration of awaited functions, even after their return, leading to wrong behaviour.

  • Python3.5+: Fix, absolute imports of names might also refer to modules and need to be handled for module loading as well.

  • Fix, the fromlist of imports could loose references, potentially leading to corruption of contained strings.

  • Python3.8: Fix, positional only arguments were not enforced to actually be that way.

  • Python3.8: Fix, complex calls with star arguments that yielded the same value twice, were not yet caught.

  • Python3.8: Fix, evaluation order for nested dictionary contractions was not followed yet.

  • Windows: Use short paths, these work much better to load extension modules and TCL parts of TkInter cannot handle unicode paths at all. This makes Nuitka work in locations, where normal Python cannot.

  • Windows: Fixup dependency walker in unicode input directories.

  • Standalone: Use frozen module loader only at libpython initialisation and switch to built-in bytecode loader that is more compatible afterwards, increasing compatibility.

  • Standalone: Fix for pydanctic support.

  • Standalone: Added missing hidden dependency of uvicorn.

  • Fix, the parser for .pyi files couldn't handle multiline imports.

  • Windows: Derive linker arch of Python from running binary, since it can happen that the Python binary is actually a script.

  • Fixup static linking with libpython.a that contains main.o by making our colliding symbols for Py_GetArgcArgv weak.

  • Python3.7: Fix misdetection as asyncgen for a normal generator, if the iterated value is async.

  • Distutils: Fix build_nuitka for modules under nested namespaces.

  • OpenBSD: Follow usage of clang and other corrections to make accelerated mode work.

  • macOS: Fixup for standalone mode library scan.

  • Fix, the logging of --show-modules was broken.

  • Windows: Enable /bigobj mode for MSVC for large compilations to work.

  • Windows: Fixup crash in warning with pefile dependency manager.

  • Windows: Fixup win32com standalone detection of other Python version win32com is in system PATH.

  • Fix, the python flag for static hashes didn't have the intended effect.

  • Fix, generators may be resurrected in the cause of their destruction, and then must not be released.

  • Fix, method objects didn't implement the methods __reduce__ and __reduce_ex__ necessary for pickling them.

  • Windows: Fix, using a Python installation through a symlink was not working.

  • Windows: Fix, icon paths that were relative were not working anymore.

  • Python3.8: Detect duplicate keywords yielded from star arguments.

  • Fix, methods could not be pickled.

  • Fix, generators, coroutines and asyncgen might be resurrected during their release, allow for that.

  • Fix, frames need to traverse their attached locals to be released in some cases.

New Features

  • Plugin command line handling now allows for proper optparse options to be used, doing away with special parameter code for plugins. The arguments now also become automatically passed to the instantiations of plugins.

    Loading and creation of plugins are now two separate phases. They are loaded when they appear on the command line and can add options in their own group, even required ones, but also with default values.

  • Started using logging with name-spaces. Applying logging per plugin to make it easier to recognize which plugin said what. Warnings are now colored in red.

  • Python3.5+: Added support for two step module loading, making Nuitka loading even more compatible.

  • Enhanced import tracing to work on standalone binaries in a useful manner, allow to compare with normal binaries.

  • Fix, the setattr built-in was leaking a reference to the None value.

Optimization

  • Proper loop SSA capable of detecting shapes with an incremental initial phase and a final result of alternatives for variables written in the loop. This detects shapes of manual integer incrementing loops correctly now, it doesn't see through iterators yet, but this will come too.

  • Added type shapes for all operations and all important built-in types to allow more compile time optimization and better target type selection.

  • Target type code generation was expanded from manual usage with conditions to all operations allowing to get at bool target values more directly.

  • For in-place operations, there is the infrastructure to generate them for improved performance, but so far it's only used for Python2 int, and not for the many types normal operations are supported.

  • Force usage of C boolean type for all indicator variables from the re-formulation. In some cases, we are not yet there with detections, and this gives instant benefit.

  • Complex constants didn't annotate their type shape, preventing compile time optimization for them.

  • Python3.8: Also support vectorcall for compiled method objects. These are rarely used in new Python, but can make a difference.

  • Remove loops that have only a final break. This happens in static optimization in some cases, and allows more optimization to be done.

  • Avoid using a preparing a constant tuple value for calls with only constant arguments.

  • Avoid using PyErr_Format where it's not necessary by adding specialized helpers for common cases.

  • Detect del statements that will raise an exception and replace with that.

  • Exception matching is boolean shape, allowing for faster code generation.

  • Disable recursion checks outside of full compat mode.

  • Avoid large blocks for conditional statements that only need to enclose the condition evaluation.

  • Added shortcuts for interactions between compiled generator variants, to avoid calls to their C methods with argument passing, etc.

Organisational

  • Updated developer manual with changes that happened, remvoing the obsolete language choice section.

  • Added 3.8 support mentions is even more places.

  • The mailing list has been deleted. We now prefer Gitter chat and Github issues for discussions.

  • Visual Code recommended extensions are now defined as such in the project configuration and you will be prompted to install them.

  • Visual Code environents for Py38 and Py27 were added for easier switch.

  • Catch usage of Python from the Microsoft App Store, it is not supported and seems to limit access to the Python installation for security reasons that make support impossible.

  • Make it clear that --full-compat should not be used in help output.

  • Added instructions for MSVC runtimes and standalone compilation to support Windows 7.

  • More complete listing of copyright holders for Debian.

  • Updated to newer black and PyLint.

  • Enhanced gcc version check, properly works with gcc 10 and higher.

Tests

  • Pylint cleanups for some of the tests.

  • Added test for loading of user plugins.

  • Removed useless outputs for search mode skipping non-matches.

Cleanups

  • Limit command line handling for multiprocessing module to when the plugin is actually used, avoiding useless code of Windows binaries.

  • Pylint cleanup also foreign code like oset and odict.

  • In preparation of deprecating the alternative, --plugin-enable has become the only form used in documentation and tests.

  • Avoid numeric pylint symbols more often.

  • Distutils: Cleanup module name for distutils commands, these are not actually enforced by distutils, but very ugly in our coding conventions.

  • The "cannot get here" code to mark unreachable code has been improved and no longer needs an identifier passed, but uses the standard C mechanism for that.

  • Removed accessors for lookup sources from nodes, allowing for faster usage and making sure, lookups are only done where needed.

Summary

This release is huge in terms of bugs fixed, but also extremely important, because the new loop SSA and type tracing, allows for many more specialized code usages. We now can trace the type for some loops to be specifically an integer or long value only, and will become able to generate code that avoids using Python objects, in these cases.

Once that happens, the performance will make a big jump. Future releases will have to consolidate the current state, but it is expected that at least an experimental addition of C type float or C long can be added, add to that iterator type shape and value analsis, and an actual jump in performance can be expected.

Nuitka Release 0.6.7

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains bug fixes and improvements to the packaging, for the RPM side as well as for Debian, to cover Python3 only systems as they are now becoming more common.

Bug Fixes

  • Compatibility: The value of __module__ for extension modules was not dependent into which package the module was loaded, it now is.

  • Anaconda: Enhanced detection of Anaconda for Python 3.6 and higher.

  • CentOS6: Detect gcc version to allow saving on macro memory usage, very old gcc didn't have that.

  • Include Python3 for all Fedora versions where it works as well as for openSUSE versions 15 and higher.

  • Windows: Using short path names to interact with Scons avoids problems with unicode paths in all cases.

  • macOS: The usage of install_name_tool could sometimes fail due to length limits, we now increase it at link time.

  • macOS: Do not link against libpython for module mode. This prevented extension modules from actually being usable.

  • Python3.6: Follow coroutine fixes in our asyncgen implementation as well.

  • Fix, our version number handling could overflow with minor versions past 10, so we limited it for now.

New Features

  • Added support for Python 3.8, the experimental was already there and pretty good, but now added the last obscure features too.

  • Plugins can now provide C code to be included in the compilation.

  • Distutils: Added targets build_nuitka and install_nuitka to complement bdist_nuitka, so we support software other than wheels, e.g. RPM packaging that compiles with Nuitka.

  • Added support for lldb the Clang debugger with the --debugger mode.

Optimization

  • Make the file prefix map actually work for gcc and clang, and compile files inside the build folder, unless we are running in debugger mode, so we use ccache caching across different compilations for at least the static parts.

  • Avoid compilation of __frozen.c in accelerated mode, it's not used.

  • Prefer using the inline copy of scons over systems scons. The later will only be slower. Use the fallback to external scons only from the Debian packages, since there we consider it forbidden to include software as a duplicate.

Organisational

  • Added recommended plugins for Visual Code, replacing the list in the Developer Manual.

  • Added repository for Fedora 30 for download.

  • Added repository for CentOS 8 for download.

  • Updated inline copy of Scons used for Python3 to 3.1.2, which is said to be faster for large compilations.

  • Removed Eclipse setup from the manual, it's only infererior at this point and we do not use it ourselves.

  • Debian: Stop recommending PyQt5 in the package, we no longer use it for built-in GUI that was removed.

  • Debian: Bumped the standards version and modernized the packaging, solving a few warnings during the build.

Cleanups

  • Scons: Avoid to add Unix only include paths on Windows.

  • Scons: Have the static source code in a dedicated folder for clarity.

Tests

  • Added tests to Github Actions, for the supported Python versions for all of Linux, macOS and Windows, covering the later publicly for the first time. We use Anaconda on macOS for the tests now, rather than Homebrew.

  • Enable IO encoding to make sure we use UTF8 for more test suites that actually need it in case of problems.

  • Comparing module outputs now handles segfaults by running in the debugger too.

Summary

This release adds full support for Python 3.8 finally, which took us a while, and it cleans up a lot on the packaging side. There aren't that many important bug fixes, but it's still nice to this cleaned up.

We have important actual optimization in the pipeline that will apply specialization to target types and for comparison operations. We expect to see actual performance improvements in the next release again.

Nuitka Release 0.6.6

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains huge amounts of crucial bug fixes all across the board. There is also new optimization and many organisational improvements.

Bug Fixes

  • Fix, the top level module must not be bytecode. Otherwise we end up violating the requirement for an entry point on the C level.

  • Fix, avoid optimizing calls with default values used. This is not yet working and needed to be disabled for now.

  • Python3: Fix, missing keyword only arguments were not enforced to be provided keyword only, and were not giving the compatible error message when missing.

  • Windows: Find win32com DLLs too, even if they live in sub folders of site-packages, and otherwise not found. They are used by other DLLs that are found.

  • Standalone: Fixup for problem with standard library module in most recent Anaconda versions.

  • Scons: Fix, was using CXXFLAGS and CPPFLAGS even for the C compiler, which is wrong, and could lead to compilation errors.

  • Windows: Make --clang limited to clang-cl.exe as using it inside a MinGW64 is not currently supported.

  • Standalone: Added support for using lib2to2.pgen.

  • Standalone: Added paths used by openSUSE to the Tcl/Tk plugin.

  • Python3.6+: Fix, the __main__ package was None, but should be "" which allows relative imports from itself.

  • Python2: Fix, compile time optimization of floor division was using normal division.

  • Python3: Fix, some run time operations with known type shapes, were falsely reporting error message with unicode or long, which is of course not compatible.

  • Fix, was caching parent package, but these could be replaced e.g. due to bytecode demotion later, causing crashes during their optimization.

  • Fix, the value of __compiled__ could be corrupted when being deleted, which some modules wrappers do.

  • Fix, the value of __package__ could be corrupted when being deleted.

  • Scons: Make sure we can always output the compiler output, even if it has a broken encoding. This should resolve MSVC issues on non-English systems, e.g. German or Chinese.

  • Standalone: Support for newest sklearn was added.

  • macOS: Added resolver for run time variables in otool output, that gets PyQt5 to work on it again.

  • Fix, floor division of run time calculations with float values should not result in int, but float values instead.

  • Standalone: Enhanced support for boto3 data files.

  • Standalone: Added support for osgeo and gdal.

  • Windows: Fix, there were issues with spurious errors attaching the constants blob to the binary due to incorrect C types provided.

  • Distutils: Fix, need to allow / as separator for package names too.

  • Python3.6+: Fix reference losses in asyncgen when throwing exceptions into them.

  • Standalone: Added support for dill.

  • Standalone: Added support for scikit-image and skimage.

  • Standalone: Added support for weasyprint.

  • Standalone: Added support for dask.

  • Standalone: Added support for pendulum.

  • Standalone: Added support for pytz and pytzdata.

  • Fix, --python-flags=no_docstrings no longer implies disabling the assertions.

New Features

  • Added experimental support for Python 3.8, there is only very few things missing for full support.

  • Distutils: Added support for packages that are in a namespace and not just top level.

  • Distutils: Added support for single modules, not only packages, by supporting py_modules as well.

  • Distutils: Added support for distinct namespaces.

  • Windows: Compare Python and C compiler architecture for MSVC too, and catch the most common user error of mixing 32 and 64 bits.

  • Scons: Output variables used from the outside, so the debugging is easier.

  • Windows: Detect if clang installed inside MSVC automatically and use it if requested via --clang option. This is only the 32 bits variant, but currently the easy way to use it on Windows with Nuitka.

Optimization

  • Loop variables were analysed, but results were only available on the inside of the loop, preventing many optimization in these cases.

  • Added optimization for the abs built-in, which is also a numerical operator.

  • Added optimization for the all built-in, adding a new concept of iteration handle, for efficient checking that avoids looking at very large sequences, of which properties can still be known.

    all(range(1, 100000))  # no need to look at all of them
    
  • Added support for optimizing ImportError construction with keyword-only arguments. Previously only used without these were optimized.

    raise ImportError(path="lala", name="lele")  # now optimized
    
  • Added manual specialization for single argument calls, sovling a TODO, as these will be very frequent.

  • Memory: Use single child form of node class where possible, the general class now raises an error if used with used with only one child name, this will use less memory at compile time.

  • Memory: Avoid list for non-local declarations in every function, these are very rare, only have it if absolutely necessary.

  • Generate more compact code for potential NameError exceptions being raised. These are very frequent, so this improves scalability with large files.

  • Python2: Annotate comparison of None with int and str types as not raising an exception.

  • Shared empty body functions and generators.

    One shared implementation for all empty functions removes that burden from the C compiler, and from the CPU instruction cache. All the shared C code does is to release its arguments, or to return an empty generator function in case of generator.

  • Memory: Added support for automatic releases of parameter variables from the node tree. These are normally released in a try finally block, however, this is now handled during code generation for much more compact C code generated.

  • Added specialization for int and long operations %, <<, >>, |, &, ^, **, @.

  • Added dedicated nodes for representing and optimizing based on shapes for all binary operations.

  • Disable gcc macro tracing unless in debug mode, to save memory during the C compilation.

  • Restored Python2 fast path for int with unknown object types, restoring performance for these.

Cleanups

  • Use dedicated ModuleName type that makes the tests that check if a given module name is inside a namespace as methods. This was hard to get right and as a result, adopting this fixed a few bugs and or inconsistent results.

  • Expand the use of nuitka.PostProcessing to cover all actions needed to get a runnable binary. This includes using install_name_tool on macOS standalone, as well copying the Python DLL for acceleration mode, cleaning the x bit for module mode. Previously only a part of these lived there.

  • Avoid including the definitions of dynamically created helper functions in the C code, instead just statically declare the ones expected to be there. This resolves Visual Code complaining about it, and should make life also easier for the compiler and caches like ccache.

  • Create more helper code in closer form to what clang-format does, so they are easier to compare to the static forms. We often create hard coded variants for few arguments of call functions, and generate them for many argument variations.

  • Moved setter/getter methods for Nuitka nodes consistently to the start of the node class definitions.

  • Generate C code much closer to what clang-format would change it to be.

  • Unified calling install_name_tool on macOS into one function that takes care of all the things, including e.g. making the file writable.

  • Debug output from scons should be more consistent and complete now.

  • Sort files for compilation in scons for better reproducible results.

  • Create code objects version independent, avoiding python version checks by pre-processor, hiding new stuff behind macros, that ignore things on older Python versions.

Tests

  • Added many more built-in tests for increased coverage of the newly covered ones, some of them being generic tests that allow to test all built-ins with typical uses.

  • Many tests have become more PyLint clean as a result of work with Visual Code and it complaining about them.

  • Added test to check PyPI health of top 50 packages. This is a major GSoC 2019 result.

  • Output the standalone directory contents for Windows too in case of a failure.

  • Added generated tests to fully cover operations on different type shapes and their errors as well as results for typical values.

  • Added support for testing against installed version of Nuitka.

  • Cleanup up tests, merging those for only Python 3.2 with 3.3 as we no longer support that version anyway.

  • Execute the Python3 tests for macOS on Travis too.

Organisational

  • The donation sponsored machine called donatix had to be replaced due to hardware breakage. It was replaced with a Raspberry-Pi 4.

  • Enhanced plugin documentation.

  • Added description of the git workflow to the Developer Manual.

  • Added checker script check-nuitka-with-codespell that reports typos in the source code for easier use of codespell with Nuitka.

  • Use newest PyLint and clang-format.

  • Also check plugin documentation files for ReST errors.

  • Much enhanced support for Visual Code configuration.

  • Trigger module code is now written into the build directory in debug mode, to aid debugging.

  • Added deep check function that descends into tuples to check their elements too.

Summary

This release comes after a long time of 4 months without a release, and has accumulated massive amounts of changes. The work on CPython 3.8 is not yet complete, and the performance work has yet to show actual fruit, but has also progressed on all fronts. Connecting the dots and pieces seems not far away.

bksahu's GSoC warp up

Hello everyone!

GSoC 2019 has almost come to an end! It's the time to wrap up this mega event started back in May 2019. Under the mentorship of Mentor Hayen, my learning experience has undergone a roller-coaster ride and it has not only boosted my growth as a developer but also as an individual. Over the last 3 months the following are my major contributions to this project:

Built-ins Optimizations

  • "any": PR #246
    • nuitka.nodes.BuiltinAnyNodes node added to optimize the "any" built-in.

    • Developed an algorithm to predict the "any" for arguments having repetitive values at compile time. For example:

      any([0]*255) -> False

      any([False, False, True]) -> True

    • Extended support for range, set and dict built-ins.

    • Added the optimized C side support too

    • Added a method getMetaClassBase to make Python 2 and Python 3 compatible while working with metaclasses.

  • Issue reported and closed Issue #349
    • Created a new module nuitka.nodes.IterationHandles to work with iterables.

    • Added support of Iteration for non-mutable types.

  • "all": PR #407
    • Added nuitka.nodes.BuiltinAllNodes to optimize the "all" built-ins.

    • Developed an algorithm similar to "any" to predict the "all" arguments. For example:

      all([0, 0, 1]) -> False

      all([True]*100) -> True

    • Other similar optimizations are done like "any" built-in.

    • Additionally, added a new testing module tests.optimizations.CommonOptimizations to test the built-ins optimizations at the same place.

  • "abs": PR #419
    • Added new operation node ExpressionOperationAbs to optimize the abs built-in.

    • Manually added shapeSlotAbs to different shapes.

    • Finally pre-computed the compile time constant abs

  • "max" and "min": PR #442
    • This PR is work in progress and is half complete.

    • This is the first optimizations in which I used reformulations instead of added in a new node.

    • Pseudo-code of "min" reformulation:

      def _min(a, b, c, ...):
       tmp_arg1 = a
       tmp_arg2 = b
       tmp_arg3 = c
       ...
       result = tmp_arg1
       if keyfunc is None: # can be decided during re-formulation
           tmp_key_result = keyfunc(result)
           tmp_key_candidate = keyfunc(tmp_arg2)
           if tmp_key_candidate < tmp_key_result:
               result = tmp_arg2
               tmp_key_result = tmp_key_candidate
           tmp_key_candidate = keyfunc(tmp_arg3)
           if tmp_key_candidate < tmp_key_result:
               result = tmp_arg3
               tmp_key_result = tmp_key_candidate
           ...
       else:
           if tmp_arg2 < result:
               result = tmp_arg2
           if tmp_arg3 < result:
               result = tmp_arg3
           ...
        return result
      
    • Adding support for keyfunc is pending

  • "zip": PR #462
    • This built-in uses both types of optimizations that the previous built-ins optimizations used.

    • zip for Python 2 uses the reformulations.

    • Pseudo-code of "zip" reformulation:

    def _zip(a, b, c, ... ):
    # First assign, to preserve order of execution,
    # the arguments might be complex expressions.
    tmp_arg1 = a
    tmp_arg2 = b
    tmp_arg3 = c
    ...
    tmp_iter_1 = iter(tmp_arg1)
    tmp_iter_2 = iter(tmp_arg2)
    tmp_iter_3 = iter(tmp_arg3)
    ...
    # could be more
    tmp_result = []
    try:
        while 1:
            tmp_result.append(
                (
                     next(tmp_iter_1),
                     next(tmp_iter_2),
                     next(tmp_iter_3),
                     ...
                )
             )
       except StopIteration:
           pass
     return tmp_result
    
    • zip for Python 3 needs a new node that calls the zip because unlike zip in Python 2, zip in Python 3 returns a zipobject.

Test suite

  • Search mode "All": PR #378
    • In the test suite, I added a new search mode "all" that will test all the modules and return all the results at once. For example:

    $ ./tests/basics/run_all.py all
       Using concrete python 2.7.12 on x86_64
       Comparing output of 'Asserts.py' using '/usr/bin/python' with flags silent, expect_success, remove_output,     recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ...
       .
       .
       .
       .
       Total 0 error(s) found.
    
  • Search mode "Only": PR #333
    • Added "only" search mode to test only a single module. For example:

    $ ./tests/basics/run_all.py only BuiltinsTest.py
       Using concrete python 2.7.12 on x86_64
       Skipping Asserts.py
       Skipping Assignments.py
       Skipping BigConstants.py
       Skipping Branching.py
       Skipping BuiltinOverload.py
       Skipping BuiltinSuper.py
       Comparing output of 'BuiltinsTest.py' using '/usr/bin/python' with flags silent, expect_success, remove_output, recurse_all, original_file, cpython_cache, plugin_enable:pylint-warnings ...
    
  • Reported and closed Issue #334: PR #336
    • Fixed the autoformat bug by reading and writing the files in bytes instead of string.

Documentation

And other minor doc fixes are added with their respective pull requests.

What I learned

  • Learned the software engineering principles and how to keep my work clean.

  • I also learned how to effectively use software designing principles like DRY and KISS.

  • Got exposed to Nuitka internals which helped me to better understand how compilers in general work.

  • Explored how CPython works internally.

  • Got some great advice from Mentor Hayen about starting my professional career in Software engineering.

Overall, it was a great experience to be a part of Nuitka :)

Yours,

Tommy's Google Summer of Code Summary

Intro

As Google Summer of Code (GSoC) is coming to an end, I am writing this blog post as a final summary describing all the work I have done as well as my experiences in this program.

Summary of My Work

  • #314 run_all.py new special-comment mechanism & Urllib3Using.py

    • Before GSoC started, I looked around for whatever work I could help with.

    • In this pull request, I added a checkRequirements function for the Nuitka standalone test suite.

    • This function checks for special-comments at the top of standalone tests in the format of # nuitka-skip-unless-expression: expression to be evaluated OR # nuitka-skip-unless-imports: module1,module2,... and will decide whether to skip a test depending on if its specified requirements are met.

    • In addition, standalone test Urllib3Using.py was created.

    • This pull request was soon merged and allowed me the lucky opportunity of GSoC 2019 with Nuitka :)

  • #339 Standalone tests for botocore & boto3 + fix to Urllib3Using.py

    • This PR was also created before the start of GSoC.

    • Standalone test Boto3Using.py was created using moto to mock AWS calls which did not turn out well.

    • Changed Urllib3Using.py with the addition of python version checks as a fix to Issue #373.

  • Urllib3 Wheel with Nuitka Pytest Results and Python-Dateutil Wheel with Nuitka Pytest Results

    • At the start of GSoC, I performed manual pytest comparison for PyPI packages urllib3 and dateutil.

    • The findings of my testing were documented in these postings.

    • Manual testing compares the pytest results of an installed nuitka wheel built using python setup.py bdist_nuitka to the regular pytest results of each package.

    • Testing is done to ensure that nuitka is building the wheel correctly.

    • If the pytests pass/fail in the same way, that means Nuitka built the wheel properly.

    • Else if the tests differ, then something is wrong.

    • Virtualenv is used to create a clean environment with no outside pollution.

    • Over the course of performing manual testing, I became familiar with the use of virtualenv, wheel, and pytest.

    • A bug was found with the package urllib3 bdist and I created Issue #413 to document the bug.

  • #440 Automating PyPI Wheel Pytest

    • After familiarizing myself with how virtualenv, wheel, and pytest work, I started to work on a script which would automate the pytest comparison for top PyPI packages.

    • The script first uses git to update each package if it is already existing in the local cache, else it will git clone that package into the local cache.

    • The script then uses calls to os.system to automate the creation of a virtualenv which is then used to install pytest and pip install the package's requirements (if any) for running pytest.

    • The script then handles each package depending on different needs before building a regular wheel with python setup.py bdist_wheel.

    • This wheel is then installed into the virtualenv, after which subprocess.Popen is used to run and capture the output of python -m pytest --disable-warnings into a string.

    • The script then resets the package to its original state and builds a nuitka-compiled wheel using python setup.py bdist_nuitka.

    • This compiled wheel is then installed into the virtualenv, after which subprocess.Popen is used to run and capture the output of python -m pytest --disable-warnings into another string.

    • The two strings containing pytest outputs are then compared to find differences.

    • If no differences are found, this means bdist_nuitka worked properly. Else Nuitka compilation did something wrong.

    • The above process is repeated for each suitable PyPI package from the PyPI top 50. (Some packages are left out if they do not contain a test suite or if they do not need to be tested)

    • At the end, a colored summary is given for all the packages tested.

    • This automation script is meant to be run regularly to inform developers of Nuitka regressions.

  • Issue #477 Unable to compile modules listed under unworthy_namespaces

    • Raised due to package pycparser failing in the automated test suite.

    • This issue will be addressed in the future.

  • Issue #479 bdist_nuitka fails for packages containing py_modules only

    • While I worked on #440, I found a bug with bdist_nuitka failing on PyPI packages containing py_modules only.

    • This bug occurs due to Nuitka making the assumption that a main package always exists for all packages. However, some packages contain only a main module and not a main package.

    • Applies to PyPI packages decorator, ipaddress, and pyparsing.

  • #483 Add support for py_modules_only compilation

    • This pull request changes bdist_nuitka.py and various other files to fix Issue #479.

    • Checks are added for the bdist_nuitka command to see if a main package exists. If there is not a main package, it will set its compile target to the main module instead.

    • This also addressed the case of a package with both a main package and a main module, in which case both are included inside the resulting wheel.

    • In addition, distutils examples py_modules_only and package_and_module were created and added for future testing.

    • During this PR, I found an import bug in Nuitka and hotfixed it with #487 Fixup_import_module.

  • #484 PyPI Standalone Tests

    • This pull request adds more standalone tests for each top PyPI package.

  • #495 Improve pypi automation

    • Improves the PyPI test suite created in #440 with functional improvements, readability improvements, and added documentation.

Things I Learned

Before GSoC, I was very uncomfortable with working inside a terminal. I was unfamiliar with many basic bash commands because I simply did not have any prior professional industrial experiences. I was also very unfamiliar with the Git flow, which is evident in the messy commit histories of my earliest pull requests.

As I continued throughout my GSoC journey, however, I became much more comfortable with working inside the terminal as well as using git as a version-control system (shoutout to my mentor Kay Hayen for helping me through all the annoying conflicts).

Although I am still no expert, I can confidently say that I am now far more proficient working with git and inside the terminal.

In addition, I became much more familiar with many of the most popular PyPI packages as well as the inner workings of python, which I believe will help me go very far in my career as a software developer.

Overall, the GSoC experience was truly astounding and I am more than thankful to my mentor Kay Hayen as well as Google for making this amazing program possible.

Yours, Tommy

Nuitka Release 0.6.5

This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler. Please see the page "What is Nuitka?" for an overview.

This release contains many bug fixes all across the board. There is also new optimization and many organisational improvements.

Bug Fixes

  • Python3.4+: Fixed issues with modules that exited with an exception, that could lead to a crash, dealing with their __spec__ value.

  • Python3.4+: The __loader__ method is_package had the wrong signature.

  • Python3.6+: Fix for async with being broken with uncompiled generators.

  • Python3.5+: Fix for coroutines that got their awaited object closed behind their back, they were complaining with RuntimeError should they be closed themselves.

  • Fix, constant values None in a bool target that could not be optimized away, lead to failure during code generation.

    if x() and None:
        ...
    
  • Standalone: Added support for sha224, sha384, sha512 in crypto package.

  • Windows: The icon wasn't properly attached with MinGW64 anymore, this was a regression.

  • Windows: For compiler outputs, also attempt preferred locale to interpret outputs, so we have a better chance to not crash over MSVC error messages that are not UTF-8 compatible.

  • macOS: Handle filename collisions for generated code too, Nuitka now treats all filesystems for all OS as case insensitive for this purpose.

  • Compatibility: Added support for tolerant del in class exception handlers.

    class C:
    
        try:
            ...
        except Exception as e:
            del e
    
            # At exception handler exit, "e" is deleted if still assigned
    

    We already were compatible for functions and modules here, but due to the special nature of class variables really living in dictionaries, this was delayed. But after some other changes, it was now possible to solve this TODO.

  • Standalone: Added support for Python3 variant of Pmw.

  • Fix, the NumPy plugin now handles more installation types.

  • Fix, the qt plugin now handles multiple library paths.

  • Fix, need libm for some Anaconda variants too.

  • Fix, left over bytecode from plugins could crash the plugin loader.

  • Fix, pkgutil.iter_packages is now working for loaded packages.

New Features

  • Python3.8: Followed some of the changes and works with beta2 as a Python 3.7, but none of the new features are implemented yet.

  • Added support for Torch, Tensorflow, Gevent, Sklearn, with a new Nuitka plugin.

  • Added support for "hinted" compilation, where the used modules are determined through a test run.

  • Added support for including TCL on Linux too.

Optimization

  • Added support for the any built-in. This handles a wide range of type shapes and constant values at compile time, while also having optimized C code.

  • Generate code for some CLONG operations in preparation of eventual per expression C type selection, it then will allow to avoid objects in many instances.

  • Windows: Avoid creating link libraries for MinGW64 as these have become unnecessary is the mean time.

  • Packages: Do not export entry points for all included packages, only for the main package name it is importable as.

Organisational

  • Added support for Visual Studio 2019 as a C compiler backend.

  • Improved plugin documentation describing how to create plugins for Nuitka even better.

  • The is now a mode for running the tests called all which will execute all the tests and report their errors, and only fail at the very end. This doesn't avoid wasting CPU cycles to report that e.g. all tests are broken, but it allows to know all errors before fixing some.

  • Added repository for Fedora 30 for download.

  • Added repository for openSUSE 15.1 for download.

  • Ask people to compile hello world program in the Github issue template, because many times, they have setup problems only.

  • Visual Studio Code is now the recommended IDE and has integrated configuration to make it immediately useful.

  • Updated internal copy of Scons to 3.1.0 as it incorporates many of our patches.

  • Changed wordings for optimization to use "lowering" as the only term to describe an optimization that simplifies.

Cleanups

  • Plugins: Major refactoring of Nuitka plugin API.

  • Plugins: To locate module kind, use core Nuitka code that handles more cases.

  • The test suite runners are also now autoformatted and checked with PyLint.

  • The Scons file is now PyLint clean too.

  • Avoid build_definitions.h to be included everywhere, in that it's only used in the main program part. This makes C linter hate us much less for using a non-existent file.

Tests

  • Run the tests using Travis on macOS for Python2 too.

  • More standalone tests have been properly whitelisting to cover openSSL usage from local system.

  • Disabled PySide2 test, it's not useful to fail and ignore it.

  • Tests: Fixups for coverage testing mode.

  • Tests: Temporarily disable some checks for constants code in reflected tests as it only exposes marshal not being deterministic.

Summary

This release is huge again. Main points are compatibility fixes, esp. on the coroutine side. These have become apparently very compatible now and we might eventually focus on making them better.

Again, GSoC 2019 is also showing effects, and will definitely continue to do soin the next release.

Many use cases have been improved, and on an organizational level, the adoption of Visual Studio Code seems an huge improvement to have a well configured IDE out of the box too.

In upcoming releases, more built-ins will be optimized, and hopefully the specialization of operations will hit more and more code with more of the infrastructure getting there.

Python-Dateutil Wheel with Nuitka Pytest Results

Intro

This post compares the pytest results of dateutil to its nuitka-built .whl counterpart.

Dateutil standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the dateutil package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

The pytest results were very similar:

Regular pytests: ============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============
Nuitka wheel pytests: ============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Steps to Reproduce

  1. Clone dateutil and nuitka into a new folder

  2. Inside the dateutil folder, issue python -m pip install -r requirements-dev.txt to install its requirements.

  3. Issue python -m pytest --disable-warnings, this runs the regular pytest for dateutil.

  4. Change into the nuitka folder and issue python setup.py develop.

  5. Change back into dateutil and issue python setup.py bdist_nuitka to build the dateutil wheel using nuitka. The newly built wheel should be found in the dist folder.

  6. Use pip to uninstall the existing dateutil, then issue python -m pip install followed by the newly built .whl filename.

  7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for dateutil.

Uncompile Python

dateutil regular pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2074 items

dateutil\test\test_easter.py ........................................... [  2%]
........................................................................ [  5%]
................................................                         [  7%]
dateutil\test\test_import_star.py .                                      [  7%]
dateutil\test\test_imports.py .......................                    [  9%]
dateutil\test\test_internals.py ....                                     [  9%]
dateutil\test\test_isoparser.py ........................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 25%]
.......x...x............................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 35%]
.....................xx                                                  [ 36%]
dateutil\test\test_parser.py ........................................... [ 38%]
........................................................................ [ 42%]
........................................................................ [ 45%]
.................................xxxxxxxxxxxxxsss......                  [ 48%]
dateutil\test\test_relativedelta.py .................................... [ 49%]
.............................................                            [ 52%]
dateutil\test\test_rrule.py ............................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 75%]
................................................................x....... [ 78%]
..............                                                           [ 79%]
dateutil\test\test_tz.py ............................s...........sssssss [ 81%]
sssssssssssssssssssssssssssssssssssssssss..s............................ [ 84%]
xxx..s......................................s........................... [ 88%]
s....................................................................... [ 91%]
..........s....................................s.s....................ss [ 95%]
sssssssssssss....s..........s........................................... [ 98%]
.............                                                            [ 99%]
dateutil\test\test_utils.py .......                                      [ 99%]
dateutil\test\property\test_isoparse_prop.py .                           [ 99%]
dateutil\test\property\test_parser_prop.py ..                            [ 99%]
docs\exercises\solutions\mlk_day_rrule_solution.py .                     [100%]

============= 1977 passed, 76 skipped, 21 xfailed in 7.99 seconds =============

Compiled with Nuitka

nuitka wheel pytest:

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\Tommy\pipenv-testing\dateutil-testing\dateutil, inifile: setup.cfg
plugins: hypothesis-4.24.3, cov-2.7.1
collected 2073 items

test\test_easter.py .................................................... [  2%]
........................................................................ [  5%]
.......................................                                  [  7%]
test\test_import_star.py .                                               [  7%]
test\test_imports.py .......................                             [  9%]
test\test_internals.py ....                                              [  9%]
test\test_isoparser.py ................................................. [ 11%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
......................................................................x. [ 25%]
..x..................................................................... [ 28%]
........................................................................ [ 32%]
........................................................................ [ 35%]
............xx                                                           [ 36%]
test\test_parser.py .................................................... [ 39%]
........................................................................ [ 42%]
........................................................................ [ 46%]
........................xxxxxxxxxxxxxsss......                           [ 48%]
test\test_relativedelta.py ............................................. [ 50%]
....................................                                     [ 52%]
test\test_rrule.py ..................................................... [ 54%]
........................................................................ [ 58%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 72%]
........................................................................ [ 75%]
.......................................................x................ [ 79%]
.....                                                                    [ 79%]
test\test_tz.py ............................s...........ssssssssssssssss [ 81%]
ssssssssssssssssssssssssssssssss..s............................xxx..s... [ 85%]
...................................s...........................s........ [ 88%]
........................................................................ [ 92%]
.s....................................s.s....................sssssssssss [ 95%]
ssss....s..........s.................................................... [ 99%]
....                                                                     [ 99%]
test\test_utils.py .......                                               [ 99%]
test\property\test_isoparse_prop.py .                                    [ 99%]
test\property\test_parser_prop.py ..                                     [100%]

============= 1976 passed, 76 skipped, 21 xfailed in 7.89 seconds =============

Urllib3 Wheel with Nuitka Pytest Results

Intro

This post compares the pytest results of urllib3 to its nuitka-built .whl counterpart.

Urllib3 standalone test have already been covered. Manual testing is now done to compare the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of the urllib3 package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

At first, the urllib3 nuitka-wheel pytest was crashing because of the unsafe assumption that imports will always exist (which is not the case if exceptions are thrown). Issue 413 was filed to record and fix this bug.

After the fixes, the pytests were ran again and the results were very similar:

Regular pytests: ====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======
Nuitka wheel pytests: ====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======

The extra passes are suspicious and require more investigation into why they happen. To make that easy, we are going to fully automate the process and compare outputs with verbose pytest modes.

Steps to Reproduce

  1. Clone urllib3 and nuitka into a new folder

  2. Inside the urllib3 folder, issue python -m pip install -r dev-requirements.txt to install its requirements.

  3. Issue python -m pytest --disable-warnings, this runs the regular pytest for urllib3.

  4. Change into the nuitka folder and issue python setup.py develop.

  5. Change back into urllib3 and issue python setup.py bdist_nuitka to build the urllib3 wheel using nuitka. The newly built wheel should be found in the dist folder.

  6. Use pip to uninstall the existing urllib3, then issue python -m pip install followed by the newly built .whl filename.

  7. Issue python -m pytest --disable-warnings, this runs the nuitka-built wheel pytest for urllib3.

Uncompile Python

urllib3 regular pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py ..                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py ..............................F.F.F..                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
.................................................Uncaught exception, closing connection.
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 3 failed, 836 passed, 456 skipped, 113 warnings in 47.54 seconds =======

Compiled with Nuitka

nuitka wheel pytest:

$ python -m pytest --disable-warnings
============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-4.0.0, py-1.8.0, pluggy-0.11.0
rootdir: C:\Users\Tommy\pipenv-testing\urllib3-testing\urllib3, inifile: setup.cfg
plugins: timeout-1.3.1
collected 1295 items

test\test_collections.py ....................................s           [  2%]
test\test_compatibility.py ...                                           [  3%]
test\test_connection.py .....                                            [  3%]
test\test_connectionpool.py ............................................ [  6%]
...........................                                              [  8%]
test\test_exceptions.py .............                                    [  9%]
test\test_fields.py ...............                                      [ 11%]
test\test_filepost.py ...........                                        [ 11%]
test\test_no_ssl.py .F                                                   [ 12%]
test\test_poolmanager.py .........................                       [ 14%]
test\test_proxymanager.py ...                                            [ 14%]
test\test_queue_monkeypatch.py .                                         [ 14%]
test\test_response.py ..................sss............................. [ 18%]
...........                                                              [ 19%]
test\test_retry.py .....................................                 [ 21%]
test\test_ssl.py ...............................                         [ 24%]
test\test_util.py ...................................................... [ 28%]
........................................................................ [ 34%]
....................ss.s...s............................................ [ 39%]
............                                                             [ 40%]
test\test_wait.py ...ssssss                                              [ 41%]
test\contrib\test_pyopenssl.py sssssssssssssssssssssssssssssssssssssssss [ 44%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 49%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 55%]
ssssssssssssssssssss                                                     [ 57%]
test\contrib\test_pyopenssl_dependencies.py ss                           [ 57%]
test\contrib\test_securetransport.py sssssssssssssssssssssssssssssssssss [ 59%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 65%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 71%]
sssssssssssssssssss                                                      [ 72%]
test\contrib\test_socks.py ..................                            [ 73%]
test\with_dummyserver\test_chunked_transfer.py ........                  [ 74%]
test\with_dummyserver\test_connectionpool.py ........................... [ 76%]
.......................................                                  [ 79%]
test\with_dummyserver\test_https.py .....................s....s......... [ 82%]
........................................................................ [ 87%]
................................sssssssssssssssssssssssssssssssssss....  [ 93%]
test\with_dummyserver\test_no_ssl.py ..                                  [ 93%]
test\with_dummyserver\test_poolmanager.py ...............                [ 94%]
test\with_dummyserver\test_proxy_poolmanager.py ................         [ 95%]
test\with_dummyserver\test_socketlevel.py .............................. [ 98%]
......................                                                   [100%]
====== 1 failed, 838 passed, 456 skipped, 113 warnings in 47.59 seconds =======