Issue226

Title Problem with too complex code in idna.core module
Priority bug Status testing
Superseder Nosy List kayhayen, obeleh
Assigned To kayhayen Keywords compiler_crash

Created on 2015-08-07.19:08:17 by obeleh, last changed by kayhayen.

Messages
msg1554 (view) Author: kayhayen Date: 2015-09-15.07:45:33
So on factory there is now an automatic fallback to bytecode inclusion for too 
complex code. With this, it should work. Please try from the "factory" branch of 
the official git repository and let me know if this works for you now.

Yours,
Kay
msg1547 (view) Author: kayhayen Date: 2015-09-14.20:12:00
The latest pre-release contains a version that will include a broken version of 
the module, which has code up until it became too complex. That's bad, but you 
got a warning about too complex code. I am making it not include the module 
again (and instead including it as bytecode if I manage).

Yours,
Kay
msg1522 (view) Author: kayhayen Date: 2015-09-09.06:59:52
I have just included in factory a change that gives a warning about this, and 
then ignores the module. Which will be fine for accelerated mode, but bad for 
standalone.

Bytecode freezing will be added later.

Yours,
Kay
msg1468 (view) Author: kayhayen Date: 2015-08-23.08:34:12
It seems only for Windows, this is an issue, and for Linux and MacOS, we could just increase the 
stack limit.

Going from a fully recursive parser to a walking thing that emits in a factory pattern is 
definitely doable. But it needs to be done for optimization and code generation as well, not just 
building.

As a practical workaround, a mass operation node might be an idea. Currently there is "+" for 2 
arguments, and a+b+c is (a+b)+c internally. Assuming, that in fact this only happens with binary 
operations in real code, why not just have a node, that does more than 2 arguments, but arbitrary 
amount of arguments and operators instead.

That kind of node would have to do its evaluation more manually, much like a complete 
"computeExpressionRaw" overload as that is in current factory git branch.

It definitely wouldn't be hotfix material. I am going to defer it until after the release that 
puts out dead value elimination (not the next one).

Yours,
Kay
msg1466 (view) Author: saslanbeigi Date: 2015-08-22.16:24:40
Hi Kay,

I was also running into the recursion limit problem with idna on OS X. I tried 
"sys.setrecursionlimit(10000)" in "bin/nuitka" and this seems to solve the problem.
msg1408 (view) Author: kayhayen Date: 2015-08-11.15:02:44
So, Python as it seems, cannot recurse deeply, because the C stack used of the Python run time 
cannot grow (on Windows at least).

Can you try, if for you sys.setrecursionlimit(4000) in "bin/nuitka" or even larger values works? I 
suspect, on Linux it does work OK as stack size grows their easily.

Yours,
Kay
msg1404 (view) Author: obeleh Date: 2015-08-08.11:56:31
Hey Kay,
Thanks for looking at the issue. Nice response :)I have no clue what the solution should be nor what this code you send actually does ;)
Greetz, Sjuul

> Subject: [issue226] Problem with /usr/local/lib/python2.7/dist-packages/idna/core.py
> To: sjuulj@hotmail.com
> From: issue_tracker@nuitka.net
> Date: Fri, 7 Aug 2015 19:39:14 +0000
> 
> 
> Kay Hayen <kay.hayen@gmail.com> added the comment:
> 
> Hello there,
> 
> ah yes, we are hitting recursion error, due to absurdly complex chains of binary operations:
> 
> codepoint_classes = {
>     'PVALID': frozenset(
>         [0x2d] +
>         list(range(0x30,0x3a)) +
>         list(range(0x61,0x7b)) +
>         list(range(0xdf,0xf7)) +
>         list(range(0xf8,0x100)) +
>         [0x101] +
>         [0x103] +
>         [0x105] +
>         [0x107] +
>         [0x109] +
>         [0x10b] +
> 
> This last chain is quasi endless. And Nuitka uses Python level recursion, for every of
> the "+" elements. This kills the Nuitka Building stage, and potentially the ast module
> when it's trying to output it.
> 
> So, should we just allow to recurse to infinity then. It seems pretty innocent to do
> something like this:
> 
>     # For some very complex Python code.
>     sys.setrecursionlimit(4000)
> 
> But then, the Python crashes running Nuitka, so there is probably a hard limit there,
> and for values that don't crash, code generation is crashing, as it needs a few more
> call indirections than building.
> 
> So probably, this comes down to not building, optimizing and generating code recursive
> at all?!
> 
> Surely a scalability issue of Python and Nuitka there.
> 
> Yours,
> Kay
> 
> _______________________________________________
> Nuitka issue tracker <issue_tracker@nuitka.net>
> <http://bugs.nuitka.net/issue226>
> _______________________________________________
msg1403 (view) Author: kayhayen Date: 2015-08-07.19:39:14
Hello there,

ah yes, we are hitting recursion error, due to absurdly complex chains of binary operations:

codepoint_classes = {
    'PVALID': frozenset(
        [0x2d] +
        list(range(0x30,0x3a)) +
        list(range(0x61,0x7b)) +
        list(range(0xdf,0xf7)) +
        list(range(0xf8,0x100)) +
        [0x101] +
        [0x103] +
        [0x105] +
        [0x107] +
        [0x109] +
        [0x10b] +

This last chain is quasi endless. And Nuitka uses Python level recursion, for every of
the "+" elements. This kills the Nuitka Building stage, and potentially the ast module
when it's trying to output it.

So, should we just allow to recurse to infinity then. It seems pretty innocent to do
something like this:

    # For some very complex Python code.
    sys.setrecursionlimit(4000)

But then, the Python crashes running Nuitka, so there is probably a hard limit there,
and for values that don't crash, code generation is crashing, as it needs a few more
call indirections than building.

So probably, this comes down to not building, optimizing and generating code recursive
at all?!

Surely a scalability issue of Python and Nuitka there.

Yours,
Kay
msg1402 (view) Author: kayhayen Date: 2015-08-07.19:23:28
Hello there,

unusual for me, but I reproduced this, on Windows with Python2.7 64 bit and pip 
installed "idna" package. Very strange, seems to be a bug of the standard Python 
ast module. I don't think I have seen such before ever. But I will try and be able 
to find out, what constructs bugs it there.

Yours,
Kay
msg1401 (view) Author: obeleh Date: 2015-08-07.19:08:17
Traceback (most recent call last):
  File "/usr/bin/nuitka", line 138, in <module>
    MainControl.main()
  File "/usr/share/nuitka/nuitka/MainControl.py", line 632, in main
    filename = filename
  File "/usr/share/nuitka/nuitka/MainControl.py", line 105, in createNodeTree
    Optimization.optimize()
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 158, in 
optimize
    changed = optimizePythonModule(current_module)
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 113, in 
optimizePythonModule
    tag_set = tag_set
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 69, in 
_optimizeModulePass
    module        = module
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 831, 
in __init__
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/StatementNodes.py", line 309, in 
computeStatementsSequence
    statement = statement
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 581, 
in onStatement
    statement.computeStatement(self)
  File "/usr/share/nuitka/nuitka/nodes/AssignNodes.py", line 62, in computeStatement
    constraint_collection.onExpression( self.getAssignSource() )
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 442, 
in onExpression
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/NodeBases.py", line 985, in 
computeExpressionRaw
    expression = sub_expression
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 442, 
in onExpression
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/NodeBases.py", line 990, in 
computeExpressionRaw
    constraint_collection = constraint_collection
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 228, in 
computeExpression
    constraint_collection = constraint_collection
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 217, in 
_attemptRecursion
    module_package        = module_package
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 135, in _consider
    reason          = reason
  File "/usr/share/nuitka/nuitka/tree/Recursion.py", line 56, in recurseTo
    is_main         = False
  File "/usr/share/nuitka/nuitka/tree/Building.py", line 1133, in createModuleTree
    is_main     = is_main
  File "/usr/share/nuitka/nuitka/tree/Building.py", line 843, in buildParseTree
    source_ref = source_ref
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 162, in buildStatementsNode
    statements = buildNodeList(provider, nodes, source_ref, allow_none = True)
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 125, in buildNodeList
    entry = buildNode( provider, node, node_source_ref, allow_none )
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 112, in buildNode
    warning( "Problem at '%s' with %s." % ( source_ref, ast.dump( node ) ) )
  File "/usr/lib/python2.7/ast.py", line 110, in dump
    return _format(node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 106, in _format
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 106, in <genexpr>
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 106, in _format
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 106, in <genexpr>
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
Traceback (most recent call last):
  File "/usr/bin/nuitka", line 138, in <module>
    MainControl.main()
  File "/usr/share/nuitka/nuitka/MainControl.py", line 632, in main
    filename = filename
  File "/usr/share/nuitka/nuitka/MainControl.py", line 105, in createNodeTree
    Optimization.optimize()
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 158, in 
optimize
    changed = optimizePythonModule(current_module)
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 113, in 
optimizePythonModule
    tag_set = tag_set
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 69, in 
_optimizeModulePass
    module        = module
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 831, 
in __init__
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/StatementNodes.py", line 309, in 
computeStatementsSequence
    statement = statement
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 581, 
in onStatement
    statement.computeStatement(self)
  File "/usr/share/nuitka/nuitka/nodes/AssignNodes.py", line 62, in computeStatement
    constraint_collection.onExpression( self.getAssignSource() )
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 442, 
in onExpression
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/NodeBases.py", line 985, in 
computeExpressionRaw
    expression = sub_expression
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 442, 
in onExpression
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/NodeBases.py", line 990, in 
computeExpressionRaw
    constraint_collection = constraint_collection
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 228, in 
computeExpression
    constraint_collection = constraint_collection
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 217, in 
_attemptRecursion
    module_package        = module_package
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 135, in _consider
    reason          = reason
  File "/usr/share/nuitka/nuitka/tree/Recursion.py", line 56, in recurseTo
    is_main         = False
  File "/usr/share/nuitka/nuitka/tree/Building.py", line 1133, in createModuleTree
    is_main     = is_main
  File "/usr/share/nuitka/nuitka/tree/Building.py", line 843, in buildParseTree
    source_ref = source_ref
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 162, in buildStatementsNode
    statements = buildNodeList(provider, nodes, source_ref, allow_none = True)
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 125, in buildNodeList
    entry = buildNode( provider, node, node_source_ref, allow_none )
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 112, in buildNode
    warning( "Problem at '%s' with %s." % ( source_ref, ast.dump( node ) ) )
  File "/usr/lib/python2.7/ast.py", line 110, in dump
    return _format(node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 106, in _format
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 106, in <genexpr>
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 106, in _format
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 106, in <genexpr>
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
Traceback (most recent call last):
  File "/usr/bin/nuitka", line 138, in <module>
    MainControl.main()
  File "/usr/share/nuitka/nuitka/MainControl.py", line 632, in main
    filename = filename
  File "/usr/share/nuitka/nuitka/MainControl.py", line 105, in createNodeTree
    Optimization.optimize()
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 158, in 
optimize
    changed = optimizePythonModule(current_module)
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 113, in 
optimizePythonModule
    tag_set = tag_set
  File "/usr/share/nuitka/nuitka/optimizations/Optimization.py", line 69, in 
_optimizeModulePass
    module        = module
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 831, 
in __init__
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/StatementNodes.py", line 309, in 
computeStatementsSequence
    statement = statement
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 581, 
in onStatement
    statement.computeStatement(self)
  File "/usr/share/nuitka/nuitka/nodes/AssignNodes.py", line 62, in computeStatement
    constraint_collection.onExpression( self.getAssignSource() )
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 442, 
in onExpression
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/NodeBases.py", line 985, in 
computeExpressionRaw
    expression = sub_expression
  File "/usr/share/nuitka/nuitka/optimizations/ConstraintCollections.py", line 442, 
in onExpression
    constraint_collection = self
  File "/usr/share/nuitka/nuitka/nodes/NodeBases.py", line 990, in 
computeExpressionRaw
    constraint_collection = constraint_collection
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 228, in 
computeExpression
    constraint_collection = constraint_collection
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 217, in 
_attemptRecursion
    module_package        = module_package
  File "/usr/share/nuitka/nuitka/nodes/ImportNodes.py", line 135, in _consider
    reason          = reason
  File "/usr/share/nuitka/nuitka/tree/Recursion.py", line 56, in recurseTo
    is_main         = False
  File "/usr/share/nuitka/nuitka/tree/Building.py", line 1133, in createModuleTree
    is_main     = is_main
  File "/usr/share/nuitka/nuitka/tree/Building.py", line 843, in buildParseTree
    source_ref = source_ref
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 162, in buildStatementsNode
    statements = buildNodeList(provider, nodes, source_ref, allow_none = True)
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 125, in buildNodeList
    entry = buildNode( provider, node, node_source_ref, allow_none )
  File "/usr/share/nuitka/nuitka/tree/Helpers.py", line 112, in buildNode
    warning( "Problem at '%s' with %s." % ( source_ref, ast.dump( node ) ) )
  File "/usr/lib/python2.7/ast.py", line 110, in dump
    return _format(node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 106, in _format
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 106, in <genexpr>
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 106, in _format
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 106, in <genexpr>
    return '[%s]' % ', '.join(_format(x) for x in node)
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]
  File "/usr/lib/python2.7/ast.py", line 94, in _format
    fields = [(a, _format(b)) for a, b in iter_fields(node)]

I'm running linux mint with a self compiled python 2.7.10
History
Date User Action Args
2015-09-15 07:45:33kayhayensetmessages: + msg1554
2015-09-14 20:12:00kayhayensetstatus: deferred -> testing
messages: + msg1547
2015-09-14 20:04:13kayhayensettitle: Problem with /usr/local/lib/python2.7/dist-packages/idna/core.py -> Problem with too complex code in idna.core module
2015-09-09 06:59:52kayhayensetmessages: + msg1522
2015-08-23 08:34:12kayhayensetstatus: chatting -> deferred
messages: + msg1468
2015-08-22 16:24:40saslanbeigisetmessages: + msg1466
2015-08-11 15:02:44kayhayensetmessages: + msg1408
keyword: + compiler_crash
2015-08-08 11:56:31obelehsetmessages: + msg1404
2015-08-07 19:39:14kayhayensetmessages: + msg1403
2015-08-07 19:23:28kayhayensetstatus: unread -> chatting
assignedto: kayhayen
messages: + msg1402
nosy: + kayhayen
2015-08-07 19:18:16kayhayensettitle: Nuitka:WARNING:Problem with statement at <SourceCodeReference to /usr/local/lib/python2.7/dist-packages/idna/core.py:1>: -> Problem with /usr/local/lib/python2.7/dist-packages/idna/core.py
2015-08-07 19:08:17obelehcreate