Issue214

Title Python module: copy_reg orphan module imports in nutika
Priority bug Status done-cbb
Superseder Nosy List marc.jofre
Assigned To Keywords

Created on 2015-07-02.18:23:38 by marc.jofre, last changed by marc.jofre.

Files
File name Uploaded Type Edit Remove
copy_reg.py marc.jofre, 2015-07-03.05:06:25 text/plain
types.py marc.jofre, 2015-07-03.05:06:00 text/plain
Messages
msg1655 (view) Author: marc.jofre Date: 2015-10-25.16:41:02
Hi all,

I solved this issue by manually tracking all the implicit imports in the used 
scripts as:
import settings # Might be necessary for nuitka compilation
import wsgi # Defined in ......\mysite. Might be necessary for nuitka 
compilation
import urls# Might be necessary for nuitka compilation
import views# Might be necessary for nuitka compilation
import dbManagement# Might be necessary for nuitka compilation
import User# Might be necessary for nuitka compilation
import Project# Might be necessary for nuitka compilation
import Measure# Might be necessary for nuitka compilation
import Management# Might be necessary for nuitka compilation
import wsgiref# Might be necessary for nuitka compilation
import socket# Might be necessary for nuitka compilation
""" Special imports for webserver manage.exe runserver 8765 """
import threading# Might be necessary for nuitka compilation
import SocketServer, subprocess, sys
from threading import Thread # Might be necessary for nuitka compilation
import logging# Might be necessary for nuitka compilation
import atexit# Might be necessary for nuitka compilation
msg1377 (view) Author: marc.jofre Date: 2015-07-25.16:38:34
Hi Kay,

I have looked into monkey patching. I am adding the following lines to test your 
suggestion:

#Using Coroutines to Create Efficient, High-Concurrency Web Applications: 
several ways to deploy WSGI applications while being compatible with nuitka 
compilation in terms of multiprocessing forking. When threads are monkey patched 
in gevent, they behave as coroutines. This means that you have to explicitly 
yield control to make it possible for other coroutines to execute.
import gevent
from gevent import monkey
monkey.patch_all(thread=False, socket=False)
msg1371 (view) Author: kayhayen Date: 2015-07-25.09:21:06
Hello Marc,

the issue there is that it is currently not identifying if a module has any side effect or 
not. Of course, a module that only defines functions, that's OK, but one which would e.g. 
make changes to "sys.path", would be another story, right?

So right now, Nuitka always only import too many modules, not too little.

Eventually, Nuitka will become able of telling that the module value is not used, and then 
only does extract the side effects of a module.

Currently these things are mostly done for functions only.

I suspect that your "manage.exe" is attempting to fork a Python to run copy_reg.py, and 
then will inherit the limited Python environment, that forbids the Python program to find 
standard library stuff outside of it.

You can try and monkey patch subprocess and similar, to see where it does that. And Nuitka 
really could benefit from being able to intercept these, and to e.g. gracefully fallback 
to copy_reg.exe should it live nearby, avoiding Python. There is a very old issue about 
that I suppose.

This could be done with os.execl, subprocess, os.system, and possibly more. Can you 
investigate those?

As for orphaned modules, I agree, it would be sweet to extract only the used parts.

Yours,
Kay
msg1348 (view) Author: marc.jofre Date: 2015-07-04.05:10:14
Hi all,

I rename the issue to orphan module imports. It is quite impressive that nuitka 
can identify the python modules to be compiled and the ones that can be skipped. 
In non-standalone or standalone compilation:

Is it possible? to make the exe safe when executed where if a module was 
indicated to be imported but at the end of the day no script really makes use of 
a method of the module. Because, in large projects, the probability of having 
imports with modules that are not really used is high.

Best,

Marc
msg1347 (view) Author: marc.jofre Date: 2015-07-03.04:40:51
Hi All,

copy_reg module fails when executing the nuitka compiled exe:

D:\SixSensoCytometerWebBased\include\bin>manage.exe runserver 8765
ImportError: No module named copy_reg

D:\SixSensoCytometerWebBased\include\bin>
msg1346 (view) Author: marc.jofre Date: 2015-07-02.19:17:35
Hi all,

Initially, copy_reg is unproperly compiled or imported. copy_reg module fails at 
execution with window command prompt dump:

D:\SixSensoCytometerWebBased\include\bin>manage.exe runserver 8765
Traceback (most recent call last):
  File "D:\SixSensoCytometerWebBased\include\bin\copy_reg.py", line 7, in <modul
e>
ImportError: No module named types

D:\SixSensoCytometerWebBased\include\bin>
msg1345 (view) Author: marc.jofre Date: 2015-07-02.18:23:38
Hi all,

I am testing proper compilation and execution with scripts requiring copy_reg. 
dispatch_table

I continue testing
History
Date User Action Args
2015-10-25 16:41:02marc.jofresetstatus: chatting -> done-cbb
messages: + msg1655
2015-07-25 16:38:34marc.jofresetmessages: + msg1377
2015-07-25 09:21:06kayhayensetmessages: + msg1371
2015-07-10 09:20:43marc.jofresettitle: Python module: orphan module imports in nutika -> Python module: copy_reg orphan module imports in nutika
2015-07-04 05:10:14marc.jofresetmessages: + msg1348
title: Python module: copy_reg.dispatch_table -> Python module: orphan module imports in nutika
2015-07-03 05:06:25marc.jofresetfiles: + copy_reg.py
2015-07-03 05:06:00marc.jofresetfiles: + types.py
2015-07-03 04:40:51marc.jofresetmessages: + msg1347
2015-07-02 19:17:35marc.jofresetstatus: unread -> chatting
messages: + msg1346
2015-07-02 18:23:38marc.jofrecreate