[PEAK] PEAK-Rules and PyPy

Tue Oct 16 18:59:41 EDT 2012

On Tue, Oct 16, 2012 at 12:48 PM, Alain Poirier
<alain.poirier at net-ng.com> wrote:
> Le 16 oct. 2012 à 18:26, Marcin Tustin <marcin.tustin at gmail.com> a écrit :
>
>> Please see: http://docs.python.org/library/__builtin__.html?highlight=__builtins__
>
> Thanks Marcin. I see from the 'CPython implementation detail' note, '__builtins__'
> can be a module or a dict. Ok, so I added a check to use '__builtins__.__dict__' if
> '__builtins__' is a module and now I've got:
>
>>>>> from peak.rules import predicates
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "/private/tmp/p/site-packages/DecoratorTools-1.8-py2.7.egg/peak/util/decorators.py", line 617, in tracer
>     frm.f_locals[k] = callback(frm,k,v,old_locals)
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 276, in callback
>     register_for_class(None)
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 270, in register_for_class
>     _register_rule(f, pred, context, cls)
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 395, in _register_rule
>     rules.add(parse_rule(Dispatching(gf).engine, pred, context, cls))
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 685, in parse_rule
>     def parse_rule(engine, predicate, context, cls):
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 953, in parse_upgrade
>     predicate, context, cls
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/core.py", line 685, in parse_rule
>     def parse_rule(engine, predicate, context, cls):
>   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/predicates.py", line 606, in _parse_string
>     maybe_bind(ctx.body, bindings), expr, ctx.actiontype, ctx.sequence
> UnboundLocalError: local variable 'expr' referenced before assignment
>
> which is strange because 'expr' is just defined some lines above.
>
>> CPython implementation detail: Most modules have the name __builtins__ (note the 's') made available as part of their globals. The value of__builtins__ is normally either this module or the value of this modules’s __dict__ attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python.
>>
>> On Tue, Oct 16, 2012 at 12:22 PM, PJ Eby <pje at telecommunity.com> wrote:
>> On Tue, Oct 16, 2012 at 11:18 AM, Alain Poirier
>> <alain.poirier at net-ng.com> wrote:
>> >   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/predicates.py", line 595, in _parse_string
>> >     b = CriteriaBuilder(engine.arguments, ctx.localdict, ctx.globaldict, __builtins__)
>> >   File "/private/tmp/PEAK-Rules-0.5a1.dev-r2707/peak/rules/codegen.py", line 334, in __init__
>> >     dict([(k,self.Const(v)) for k,v in ns.iteritems()]) for ns in namespaces
>> > AttributeError: 'module' object has no attribute 'iteritems'
>> >
>> >
>> > Do you think it could be possible to have a working PyPy version of Peak Rules?
>>
>> It depends. It looks like the problem above is that __builtins__ in
>> PyPy might be a module instead of a dictionary?  If so, that's
>> probably a bug in PyPy that needs to be fixed.
>>
>> There are likely to be other problems besides this one, but let's take
>> them one at a time.  ;-)

There are bigger fish to fry - the AddOns package (which is a
dependency) fails all its tests because PyPy doesn't support
non-string keys in type dictionaries.  DecoratorTools' test suite also
fails, but AFAICT it's all due to changes in repr() of various
built-in types (unless PyPy doesn't support classic classes), and
should be shallow.  AddOns, however, is quite heavily used by
PEAK-Rules.

An important BytecodeAssembler test also fails: PyPy has a
slightly-incompatible bytecode interpreter, and PEAK-Rules abuses a
weak link in CPython's bytecode to implement a "computed goto"
operation in generated bytecode.  My guess is that the UnboundLocal
errors are being caused by how PyPy handles (or more precisely,
doesn't handle) the computed goto.

The specific issue is this: in CPython, the END_FINALLY bytecode takes
a "why" value on top of the stack, plus an extra value as a jump
offset.  PyPy, however, wants one value on the stack: a special
interpreter-owned value (that's AFAICT can't be created directly in
Python) that wraps the reason and the jump offset together.

Unless there's some way to create these special SContinueLoop objects
from Python (and I'm guessing there's not), I'd have to write a
replacement code generator for PyPy that uses a linear search instead
-- which ironically may make PyPy's predicate dispatch slower than
CPython's in some cases, unless the JIT can optimize it out enough.

[pause for some hacking]

After hacking around a bit, I have a quick and dirty patch (attached)
that makes most of the tests pass, by fixing up the __builtins__, and
handling the END_FINALLY issue by using a linear search in the
innermost dispatch loops.  It isn't suitable for release at the
moment, because it doesn't check for whether it's running under PyPy,
and there are still some tests that fail due to shallow minor issues
like the PyPy repr() differences and hash iteration differences.  Let
me know if it works for you, and maybe I'll clean it up for release.

Technically, AddOns and BytecodeAssembler need some fixes too, but the
parts of them that don't work the same under PyPy don't actually get
used by PEAK-Rules, at least not in the test suite.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: quick.patch
Type: application/octet-stream
Size: 3495 bytes
Desc: not available
Url : http://www.eby-sarna.com/pipermail/peak/attachments/20121016/637b2857/quick-0001.obj