Issue380

Title Scons: Can give strerror message.
Priority bug Status in-progress
Superseder Nosy List jmatos
Assigned To Keywords compiler

Created on 2017-10-21.21:33:25 by jmatos, last changed by jmatos.

Messages
msg2282 (view) Author: jmatos Date: 2017-10-27.01:25:37
Hello,

I spoke too soon.
I reverted your change and everyhting stills works.

Could it some caching? Is there a way to clear that caching a redo the 
tests?

Best regards,

JM

On 26-10-2017 12:57, Kay Hayen (via Issue Tracker) wrote:
> Kay Hayen <kay.hayen@gmail.com> added the comment:
>
> Can you try and make that change to: nuitka/build/inline_copy/lib/scons-3.0.0/SCons/CacheDir.py
>
> -            if os.path.isdir(path) and len(os.listdir(path)) != 0:
> +            if os.path.isdir(path) and len(f for f in os.listdir(path) if os.path.basename(f) != "config") != 0:
>
> I assume, that ought to catch that race, by ignoring empty caches as well, that other threads will have created before
> you get a chance to look.
>
> Also, bonus points, if you report the bug upstream.
>
> What could be the case, and here comes in your compiler change, is that MSVC 2017 is somehow faster than MSVC 2015 (I
> would always assume more bloated ;-) and what happens is that not only did on thread enter without the directory
> existing, but another also already populated the cache. Then my code change will not help.
>
> Yours,
> Kay
>
> _______________________________________________
> Nuitka issue tracker <issue_tracker@nuitka.net>
> <http://bugs.nuitka.net/issue380>
> _______________________________________________
msg2281 (view) Author: jmatos Date: 2017-10-26.23:47:32
Hello,

Before trying your change I tried to slow down my portable by changing 
it's power config to Balance and removing the power cable.
That way, there wasn't any error or warning in the many times I tried.
As soon as I moved the power config to Performance (or connected the 
power cable) the warnings started to appear.

After that, I made the change you requested and there were no errors or 
warnings, even after changing the power config to Performance or 
connecting the power cable.
I tried several builds to be sure, even with different Pythons (CPython 
and WinPython). It always worked fine.

I think you nailed it.

Thanks.

I can't explain the tech details to the SCONs team, so I ask you, if you 
don't mind, to report the bug to them.

Best regards,

JM

On 26-10-2017 12:57, Kay Hayen (via Issue Tracker) wrote:
> Kay Hayen <kay.hayen@gmail.com> added the comment:
>
> Can you try and make that change to: nuitka/build/inline_copy/lib/scons-3.0.0/SCons/CacheDir.py
>
> -            if os.path.isdir(path) and len(os.listdir(path)) != 0:
> +            if os.path.isdir(path) and len(f for f in os.listdir(path) if os.path.basename(f) != "config") != 0:
>
> I assume, that ought to catch that race, by ignoring empty caches as well, that other threads will have created before
> you get a chance to look.
>
> Also, bonus points, if you report the bug upstream.
>
> What could be the case, and here comes in your compiler change, is that MSVC 2017 is somehow faster than MSVC 2015 (I
> would always assume more bloated ;-) and what happens is that not only did on thread enter without the directory
> existing, but another also already populated the cache. Then my code change will not help.
>
> Yours,
> Kay
>
> _______________________________________________
> Nuitka issue tracker <issue_tracker@nuitka.net>
> <http://bugs.nuitka.net/issue380>
> _______________________________________________
msg2279 (view) Author: kayhayen Date: 2017-10-26.11:57:59
Can you try and make that change to: nuitka/build/inline_copy/lib/scons-3.0.0/SCons/CacheDir.py

-            if os.path.isdir(path) and len(os.listdir(path)) != 0:
+            if os.path.isdir(path) and len(f for f in os.listdir(path) if os.path.basename(f) != "config") != 0:

I assume, that ought to catch that race, by ignoring empty caches as well, that other threads will have created before 
you get a chance to look.

Also, bonus points, if you report the bug upstream.

What could be the case, and here comes in your compiler change, is that MSVC 2017 is somehow faster than MSVC 2015 (I 
would always assume more bloated ;-) and what happens is that not only did on thread enter without the directory 
existing, but another also already populated the cache. Then my code change will not help.

Yours,
Kay
msg2278 (view) Author: kayhayen Date: 2017-10-26.11:50:16
Hello there,

indeed, scons runs that constructor as many times as you have a CPU and on my system, it enters the path where it says this, for every there. If I 
put assert False after exists test, that raises for each thread.

        config_file = os.path.join(path, 'config')
        if not os.path.exists(config_file):
            # A note: There is a race hazard here, if two processes start and
            # attempt to create the cache directory at the same time. However,
            # python doesn't really give you the option to do exclusive file
            # creation (it doesn't even give you the option to error on opening
            # an existing file for writing...). The ordering of events here
            # as an attempt to alleviate this, on the basis that it's a pretty
            # unlikely occurence (it'd require two builds with a brand new cache
            # directory)
            if os.path.isdir(path) and len(os.listdir(path)) != 0:
                self.config['prefix_len'] = 1
                # When building the project I was testing this on, the warning
                # was output over 20 times. That seems excessive
                global warned
                if self.path not in warned:
                    msg = "Please upgrade your cache by running " +\
                          " scons-configure-cache.py " +  self.path
                    SCons.Warnings.warn(SCons.Warnings.CacheVersionWarning, msg)
                    warned[self.path] = True
            else:
                if not os.path.isdir(path):
                    try:
                        os.makedirs(path)
                    except OSError:
                        # If someone else is trying to create the directory at
                        # the same time as me, bad things will happen
                        msg = "Failed to create cache directory " + path
                        raise SCons.Errors.EnvironmentError(msg)

                self.config['prefix_len'] = 2
                if not os.path.exists(config_file):
                    try:
                        with open(config_file, 'w') as config:
                            json.dump(self.config, config)
                    except:
                        msg = "Failed to write cache configuration for " + path
                        raise SCons.Errors.EnvironmentError(msg)
        else:
            try:
                with open(config_file) as config:
                    self.config = json.load(config)
            except ValueError:
                msg = "Failed to read cache configuration for " + path
                raise SCons.Errors.EnvironmentError(msg)


So what's going on, is that apparently a re-used cache directory gets prefix_len 1 and the existing one would
get 2.

In your setup apparently, the race is that the config file is created, and thus len(os.listdir() is not empty.

Clearly a scons bug. There is no 3.0.1 yet however.

Yours,
Kay
msg2277 (view) Author: kayhayen Date: 2017-10-26.11:26:50
Hello there,

seeing the distinct cache path, including the "3.5", albeit poorly joined with the 
arch, is sufficient proof. I think we can clearly see that it is the same Python 
instance running all the time. I can no longer believe that it is a surviving cache 
directory from before.

Looking at the code, it should mean that a "config" file is not present in that cache 
directory, but the cache directory exists and is not empty.

What could be going on, is that:

a) The cache creation of Scons is not thread safe. The call stack seems to suggest it 
is being used in a thread. The code is in the cache dir constructor though, and may 
only ever run once, but maybe scons delays that work too.
b) The cleanSourceDirectory of Nuitka removes the config file. I cannot see how that 
would be possile though.

I will do a bit of debugging there, to see if it's run multiple times, and what the 
cleaning does actually clean for me.

Yours,
Kay
msg2270 (view) Author: jmatos Date: 2017-10-24.10:28:21
Hello,

After upgrading the VS2017 machine from Nuitka 0.5.28 to 0.5.28.1 it no 
longer shows those 2 messages (warning and error), but shows the same 
warning on a different file.

(dexp) C:\Users\JM\AppData\Local\Temp\DExp_build\DExp>nuitka --module 
--recurse-none --plugin-enable=pylint-warnings base_hash_checker.py
scons: warning: Please upgrade your cache by running 
scons-configure-cache.py base_hash_checker.build\cache-x863.5
File "c:\python354-32\Lib\threading.py", line 914, in _bootstrap_inner

After running again no warnings or errors appeared.

After running yet again an error appeared on another file:

C:\Users\JM\AppData\Local\Temp\DExp_build\DExp>nuitka --module 
--recurse-none --plugin-enable=pylint-warnings data_processing.py
scons: *** [data_processing.build\module.data_processing.obj] 
AttributeError : 'EnvironmentError' object has no attribute 'strerror'

After running again new warnings on different files:

(dexp) C:\Users\JM\AppData\Local\Temp\DExp_build\DExp>nuitka --module 
--recurse-none --plugin-enable=pylint-warnings base.py
scons: warning: Please upgrade your cache by running 
scons-configure-cache.py base.build\cache-x863.5
File "c:\python354-32\Lib\threading.py", line 914, in _bootstrap_inner

(dexp) C:\Users\JM\AppData\Local\Temp\DExp_build\DExp>nuitka --module 
--recurse-none --plugin-enable=pylint-warnings common.py
scons: warning: Please upgrade your cache by running 
scons-configure-cache.py common.build\cache-x863.5
File "c:\python354-32\Lib\threading.py", line 914, in _bootstrap_inner

(dexp) C:\Users\JM\AppData\Local\Temp\DExp_build\DExp>nuitka --module 
--recurse-none --plugin-enable=pylint-warnings dexp_config.py
scons: warning: Please upgrade your cache by running 
scons-configure-cache.py dexp_config.build\cache-x863.5
File "c:\python354-32\Lib\threading.py", line 914, in _bootstrap_inner

These warnings and errors seem random, but again only occur in the 
VS2017 machine.

I don't think it is worth your time to analyze my gui_tooltip.py because 
it clearly is not the origin of the scons error.

Best regards,

JM

On 24-10-2017 07:16, Kay Hayen (via Issue Tracker) wrote:
> Kay Hayen<kay.hayen@gmail.com>  added the comment:
>
> I understand that my statement is in conflict with yours. However, what scons
> wants to say is that the "cache" directory needs to be migrated. And the reason
> for the message is that I think when run with Python3 another database format is
> used than with Python2. It doesn't have to do with MSVC versions.
>
> I would be proven correct by the last release not doing anymore, and incorrect by
> it doing it.
>
> I will try to compile the gui_tooltip.py later when I am on Windows, that is the
> more interesting part of the issue of course.
>
> Yours,
> Kay
>
> _______________________________________________
> Nuitka issue tracker<issue_tracker@nuitka.net>
> <http://bugs.nuitka.net/issue380>
> _______________________________________________
msg2269 (view) Author: kayhayen Date: 2017-10-24.06:16:40
I understand that my statement is in conflict with yours. However, what scons 
wants to say is that the "cache" directory needs to be migrated. And the reason 
for the message is that I think when run with Python3 another database format is 
used than with Python2. It doesn't have to do with MSVC versions.

I would be proven correct by the last release not doing anymore, and incorrect by 
it doing it.

I will try to compile the gui_tooltip.py later when I am on Windows, that is the 
more interesting part of the issue of course.

Yours,
Kay
msg2266 (view) Author: jmatos Date: 2017-10-23.09:07:22
Hello,

I don't understand what you mean by ".. one build directory from Python3 
to Python2 execution of scons.".
The machine with VS2017 only has Python 3.

I tried Nuitka 0.5.28 with only Python 3 on another machine (which has 
VS2015 and had Python 2 but I removed it to test Nuitka) and it worked 
without any error.

The only change I see between the 2 machines is the VS version.

I will send you gui_tooltip.py using your gmail account.

Best regards,

JM

On 23-10-2017 08:42, Kay Hayen (via Issue Tracker) wrote:
> Kay Hayen <kay.hayen@gmail.com> added the comment:
>
> Your analysis is wrong, the issue is not which MSVC you are using, but changing
> on one build directory from Python3 to Python2 execution of scons. That had been
> addressed on develop at the time and is part of the recent release.
>
> The strerror issue would be interesting to reproduce. It seems however to be a
> scons issue. It might be hiding a compilation error though, as it seems it wants
> to talk about a failed compilation right there. Can you attach gui_tooltip.py or
> send it to me privately if secret, so I can try myself.
>
> Yours,
> Kay
>
> ----------
> keyword: +compiler
> status: unread -> in-progress
> title: Strange Warning and Error when using Nuitka 0.5.28 with VS 2017 (everything works fine with VS2015) -> Scons: Can give strerror message.
>
> _______________________________________________
> Nuitka issue tracker <issue_tracker@nuitka.net>
> <http://bugs.nuitka.net/issue380>
> _______________________________________________
msg2265 (view) Author: kayhayen Date: 2017-10-23.07:42:05
Your analysis is wrong, the issue is not which MSVC you are using, but changing 
on one build directory from Python3 to Python2 execution of scons. That had been 
addressed on develop at the time and is part of the recent release.

The strerror issue would be interesting to reproduce. It seems however to be a 
scons issue. It might be hiding a compilation error though, as it seems it wants 
to talk about a failed compilation right there. Can you attach gui_tooltip.py or 
send it to me privately if secret, so I can try myself.

Yours,
Kay
msg2264 (view) Author: jmatos Date: 2017-10-21.21:33:25
Hello,

I've been using Nuitka for some time and everything was working until I upgraded
one of my dev PCs to VS2017. My other machines are still with VS2015.

The following commands work with Nuitka 0.5.28 and VS2015 but return the
following messages with VS2017.

Using the command
nuitka --module --recurse-none --plugin-enable=pylint-warnings gui_toolbar.py
returns the following warning
scons: warning: Please upgrade your cache by running  scons-configure-cache.py
gui_toolbar.build\cache-x863.5
File "c:\python354-32\Lib\threading.py", line 914, in _bootstrap_inner

Using the command
nuitka --module --recurse-none --plugin-enable=pylint-warnings gui_tooltip.py
returns the following error
scons: *** [gui_tooltip.build\__constants.obj] AttributeError :
'EnvironmentError' object has no attribute 'strerror'

Tested with Windows 7 Home Premium x64 (VS2017), Windows 7 Pro x64 (VS2015) and
Windows 8.1 x64 (VS2015), all with Python 3.5.4. 32b.

Best regards,

JM
History
Date User Action Args
2017-10-27 01:25:37jmatossetmessages: + msg2282
2017-10-26 23:47:32jmatossetmessages: + msg2281
2017-10-26 11:57:59kayhayensetmessages: + msg2279
2017-10-26 11:50:16kayhayensetmessages: + msg2278
2017-10-26 11:26:50kayhayensetmessages: + msg2277
2017-10-24 10:28:21jmatossetmessages: + msg2270
2017-10-24 06:16:40kayhayensetmessages: + msg2269
2017-10-23 09:07:22jmatossetmessages: + msg2266
2017-10-23 07:42:05kayhayensetstatus: unread -> in-progress
messages: + msg2265
keyword: + compiler
title: Strange Warning and Error when using Nuitka 0.5.28 with VS 2017 (everything works fine with VS2015) -> Scons: Can give strerror message.
2017-10-21 21:33:25jmatoscreate