Issue377

Title UnicodeEncodeError in lib\subprocess.py
Priority bug Status chatting
Superseder Nosy List jer70
Assigned To jer70 Keywords

Created on 2017-09-04.15:43:48 by jer70, last changed by jer70.

Messages
msg2256 (view) Author: jer70 Date: 2017-10-17.14:27:54
Dear Kay,

Sorry for not having been responsive, I busied myself with totally other 
topics recently and was not paying enough attention to your response 
asking for some extra information.

Below is the required output section, showing use of g++ 6.3.0. (MinGW). 
To which extent is this an "incompatible compiler", and what would be a 
"compatible compiler" according to you ??

WIth best regards,

Jerome, Geneva (Switzerland)

Scons command: C:\Python2716_32\python.exe 
C:\Python2716_32\lib\site-packages\nuitka\build\inline_copy\bin\scons.py 
-f C:\Python2716_32\lib\site-packages\nuitka\build\SingleExe.scons 
--jobs 8 --warn=no-deprecated --no-site-dir --debug=explain 
show_scons=true cache_mode=true name=FSOpenLink python_debug=false 
unstripped_mode=false full_compat=false module_mode=false 
module_count=240 debug_mode=false target_arch=x86 trace_mode=false 
result_name=FSOpenLink 
nuitka_src=C:\Python2716_32\lib\site-packages\nuitka\build 
nuitka_cache=c:\work\nuitka_cache python_prefix=C:\Python2716_32 
source_dir=FSOpenLink.build experimental= python_version=2.7
scons: Reading SConscript files ...
Initial CC: gcc
Initial CCVERSION: 6.3.0
Initial CXX: g++
Initial CXXVERSION: 6.3.0
Scons compiler: Using C:\MinGW\bin\gcc.exe
scons: Told to run compilation on 8 CPUs.
scons: done reading SConscript files.
scons: Building targets ...
scons: building `FSOpenLink.build\module.appdirs.o' because it doesn't exist
Retrieved `FSOpenLink.build\module.appdirs.o' from cache
scons: building `FSOpenLink.build\module.Blocking.Blocking.o' because it 
doesn't exist
Retrieved `FSOpenLink.build\module.Blocking.Blocking.o' from cache
scons: building `FSOpenLink.build\module.Blocking.o' because it doesn't 
exist
Retrieved `FSOpenLink.build\module.Blocking.o' from cache
scons: building `FSOpenLink.build\module.comtypes.automation.o' because 
it doesn't exist
gcc -o FSOpenLink.build\module.comtypes.automation.o -c -std=c11 
-fpartial-inlining -fno-var-tracking -O3 -pipe -D_WIN32_WINNT=0x0501 
-D_NUITKA_EXPERIMENTAL_ -D_NUITKA_SYSFLAG_PY3K_WARNING=0 
-D_NUITKA_SYSFLAG_DIVISION_WARNING=0 -D_NUITKA_SYSFLAG_UNICODE=0 
-D_NUITKA_SYSFLAG_BYTES_WARNING=0 -D_NUITKA_SYSFLAG_NO_SITE=0 
-D_NUITKA_SYSFLAG_VERBOSE=0 -D__NUITKA_NO_ASSERT__ 
-D_NUITKA_CONSTANTS_FROM_RESOURCE -D_NUITKA_FROZEN=0 
-D_NUITKA_MODULE_COUNT=240 -D_NUITKA_EXE -IC:\Python2716_32\include 
-IC:\Python2716_32\PC -IFSOpenLink.build 
-IC:\Python2716_32\lib\site-packages\nuitka\build\include 
-IC:\Python2716_32\lib\site-packages\nuitka\build\static_src 
FSOpenLink.build\module.comtypes.automation.c

Le 17.10.2017 à 13:41, Kay Hayen (via Issue Tracker) a écrit :
> Kay Hayen <kay.hayen@gmail.com> added the comment:
>
> I just made the stable release with that improvement. Ought to work now. Due to
> lack of feedback about the compiler question, I am simply closing as resolved.
>
> ----------
> status: chatting -> resolved
>
> _______________________________________________
> Nuitka issue tracker <issue_tracker@nuitka.net>
> <http://bugs.nuitka.net/issue377>
> _______________________________________________
>
msg2235 (view) Author: kayhayen Date: 2017-10-17.11:41:26
I just made the stable release with that improvement. Ought to work now. Due to 
lack of feedback about the compiler question, I am simply closing as resolved.
msg2232 (view) Author: kayhayen Date: 2017-10-03.11:28:52
The cache directory path is unicode and that ought to be encoded for the subprocess 
module on at least Windows it seems.

I am changing the scons call to more properly tunnel any argument and therefore won't 
be limited to the cache directory, but also Python prefix, and source directory, which 
could contain unicode too. That should then be a generic solution to the sort of 
problem. I will shortly make a release with that.

Not having the uintptr_t indicates that you are using an incompatible compiler. Can you 
provide the output with "--show-scons" option, specifically where it talks about what 
compiler it selected.

Yours,
Kay
msg2224 (view) Author: jer70 Date: 2017-09-05.15:38:48
Ok, I have been able to redirect the cache directory to my c:\work by replacing
the call to getCacheDir() by a hardcoded path. It solved the problem.

But now here comes the next one :

In file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.Blocking.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
scons: *** [FSOpenLink.build\module.Blocking.o] Error 1
In file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.Blocking.Blocking.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
scons: *** [FSOpenLink.build\module.Blocking.Blocking.o] Error 1
In file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.appdirs.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
scons: *** [FSOpenLink.build\module.appdirs.o] Error 1
In file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.comtypes.client.lazybind.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
scons: *** [FSOpenLink.build\module.comtypes.client.lazybind.o] Error 1
In file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.comtypes.client.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
Isn file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.comtypes.client.dynamic.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
cons: *** [FSOpenLink.build\module.comtypes.client.o] Error 1
scons: *** [FSOpenLink.build\module.comtypes.client.dynamic.o] Error 1
In file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.comtypes.automation.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
scons: *** [FSOpenLink.build\module.comtypes.automation.o] Error 1
In file included from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/compiled_generator.h:30:0,
                 from
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/prelude.h:244,
                 from FSOpenLink.build\module.comtypes.c:19:
C:\Python2716_32\lib\site-packages\nuitka\build\include/nuitka/fibers.h:56:43:
erreur: unknown type name 'uintptr_t'
 int _prepareFiber( Fiber *to, void *code, uintptr_t arg );
                                           ^
scons: *** [FSOpenLink.build\module.comtypes.o] Error 1
msg2223 (view) Author: jer70 Date: 2017-09-05.13:09:47
Ok, then here is the requested output :

Scons_command is : ['c:\\Python2716_32\\python.exe',
'c:\\Python2716_32\\lib\\site-packages\\nuitka\\build\\inline_copy\\bin\\scons.py',
'--quiet', '-f',
'c:\\Python2716_32\\lib\\site-packages\\nuitka\\build\\SingleExe.scons',
'--jobs', '8', '--warn=no-deprecated', '--no-site-dir', 'cache_mode=true',
'name=FSOpenLink', 'python_debug=false', 'unstripped_mode=false',
'full_compat=false', 'module_mode=false', 'module_count=235',
'debug_mode=false', 'target_arch=x86', 'trace_mode=false',
'result_name=FSOpenLink',
'nuitka_src=c:\\Python2716_32\\lib\\site-packages\\nuitka\\build',
u'nuitka_cache=C:\\Users\\J\xe9r\xf4me\\AppData\\Local\\Nuitka\\Nuitka\\Cache',
'python_prefix=c:\\Python2716_32', 'source_dir=FSOpenLink.build',
'experimental=', 'python_version=2.7']

I suppose the problem comes from the fact that Nuitka tries to create and use a
cache directory in my home directory, whose path contains 2 non-ascii characters.
Possible solutions :
a) (Preferred) : redirect the default Nuitka cache location. Can you tell me how
to do it ?
b) Rename my user account / homedir.
msg2222 (view) Author: kayhayen Date: 2017-09-05.09:09:29
I would assume you cannot compile a hello world program. Please edit 
c:\Python2716_32\lib\site-packages\nuitka\build\SconsInterface.py and make it output the 
scons_command by putting a print statement there.
msg2221 (view) Author: jer70 Date: 2017-09-04.15:43:46
Python 2.7.13 (32bits) on Win10 64 bits platform
nuitka --version --> 0.5.27

I am trying to compile a rather lenghty and complex set of Python packages.
Compilation crashes in module lib\subprocess.py with following exception :

Traceback (most recent call last):
  File "c:\Python2716_32\Scripts\nuitka", line 211, in <module>
    MainControl.main()
  File "c:\Python2716_32\lib\site-packages\nuitka\MainControl.py", line 766, in main
    main_module = main_module
  File "c:\Python2716_32\lib\site-packages\nuitka\MainControl.py", line 682, in
compileTree
    quiet       = not Options.isShowScons()
  File "c:\Python2716_32\lib\site-packages\nuitka\MainControl.py", line 549, in
runScons
    return SconsInterface.runScons(options, quiet), options
  File "c:\Python2716_32\lib\site-packages\nuitka\build\SconsInterface.py", line
266, in runScons
    return subprocess.call(scons_command, shell = False) == 0
  File "c:\Python2716_32\lib\subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "c:\Python2716_32\lib\subprocess.py", line 390, in __init__
    errread, errwrite)
  File "c:\Python2716_32\lib\subprocess.py", line 640, in _execute_child
    startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position
501: ordinal not in range(128)

* * *

Simply trying to compile module subprocess.py produces the same exception :

nuitka --recurse-all c:\Python2716_32\lib\subprocess.py
-->
Traceback (most recent call last):
  File "c:\Python2716_32\Scripts\nuitka", line 211, in <module>
    MainControl.main()
  File "c:\Python2716_32\lib\site-packages\nuitka\MainControl.py", line 766, in main
    main_module = main_module
  File "c:\Python2716_32\lib\site-packages\nuitka\MainControl.py", line 682, in
compileTree
    quiet       = not Options.isShowScons()
  File "c:\Python2716_32\lib\site-packages\nuitka\MainControl.py", line 549, in
runScons
    return SconsInterface.runScons(options, quiet), options
  File "c:\Python2716_32\lib\site-packages\nuitka\build\SconsInterface.py", line
266, in runScons
    return subprocess.call(scons_command, shell = False) == 0
  File "c:\Python2716_32\lib\subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "c:\Python2716_32\lib\subprocess.py", line 390, in __init__
    errread, errwrite)
  File "c:\Python2716_32\lib\subprocess.py", line 640, in _execute_child
    startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position
501: ordinal not in range(128)
History
Date User Action Args
2017-10-17 14:27:54jer70setstatus: resolved -> chatting
messages: + msg2256
2017-10-17 11:41:26kayhayensetstatus: chatting -> resolved
messages: + msg2235
2017-10-03 11:28:52kayhayensetmessages: + msg2232
2017-09-05 15:38:48jer70setmessages: + msg2224
2017-09-05 13:09:48jer70setmessages: + msg2223
2017-09-05 09:09:39kayhayensetassignedto: jer70
2017-09-05 09:09:29kayhayensetstatus: unread -> chatting
messages: + msg2222
2017-09-04 15:43:48jer70create