[PEAK] PEAK-Rules for Python3

[Cara]

On Mon, 2015-03-23 at 00:44 -0400, PJ Eby wrote:
> To answer the earlier question, I'm probably not going to be
> interested in integrating patches (unless they are relatively small)
> as I don't expect I will be moving to Python 3 myself.  As I believe
> I've mentioned before, the key challenges to getting PEAK-Rules
> working on Python 3 are getting its dependencies working first (mainly
> DecoratorTools and BytecodeAssembler, but also AddOns and Extremes).
> I would be interested in seeing what patches are needed for those
> modules to fulfill their tests on Python 3 before I'd look at
> integrating PEAK-Rules stuff.  In general, I don't have a lot of time
> for hacking Python stuff these days, as it doesn't overlap with my
> work life very much any more.  And when it comes to hobby time, I've
> also become rather fond of Node and Coffeescript.  (Which now make
> Python look to me like an overly complicated and restrictive language
> by comparison, oddly enough.)  Anyway, I'm willing to integrate
> reasonable patches, and can offer advice, but not much in the way of
> actual development or testing.

This is a status update and also notes for anyone else who attempts this
in the future.

Before I touched the code base at all, I ran all the doctests in Python
2.7 to get an idea what was failing.  The results are here:

AddOns/README.txt: 10/94 failures.
BytecodeAssembler/README.txt: 72/755 failures.
DecoratorTools/README.txt: 15/89 failures.
Extremes/README.txt: passed.
PEAK-Rules/AST-Builder.txt: 1/138 failures.
PEAK-Rules/Code-Generation.txt 12/162 failures.
PEAK-Rules/Criteria.txt: 21/246 failures.
PEAK-Rules/DESIGN.txt: 26/153 failures.
PEAK-Rules/Indexing.txt: 11/272 failures.
PEAK-Rules/Predicates.txt: 28/208 failures.
PEAK-Rules/README.txt: 2/87 failures.
PEAK-Rules/Syntax-Matching.txt: 10/41 failures.
SymbolType/README.txt: passed.

I didn't go through these in detail and classify them all, but some of
them look like shallow errors where at some point Python started
prefixing some classes with the module name (e.g.
<class '__main__.Index'> instead of <class 'Index'>).  I couldn't find
any notes about the options I needed to pass to run the doctests, so
some of that may be bad options.

I ran futurize,
http://python-future.org/automatic_conversion.html#futurize-py2-to-py2-3
, on the main code and 2to3 -d on the doctests, then fixed some of the
obvious Python 2/3 differences like using types instead of new, removing
support for unbound methods, changing the argument count for creating
code objects, and so on.  At this point, some of the packages have most
of their functionality working: for instance, all the doctests pass for
Extremes, test_decorators.py passes, and most of the same doctests for
DecoratorTools/README.txt pass.  (The only functionality I could find
that was broken is that the class decorators don't seem to be working.)

That said, there are two obstacles going forward.

1) Breaking backwards compatibility: Python has had, for instance,
class decorators since 2.6, which isn't even supported any more.  I
don't have any interest in rewriting the frame-hacking for
DecoratorTools.decorator_class() to make it work on Python 3, so I'd
just refactor all the relevant code to use the built-in class
decorators.  I assume that means I'm now outside the scope of acceptable
patches.  The AST builder and other parts of DecoratorTools have the
same status.

2) Implicit relative imports: most of the packages are filled with lines
like,
from peak.utils.decorators import ...
In Python 3, these don't work and the automatic conversion tools don't
work on them.  They're responsible for most of the failures I'm seeing
in the doctests.  I could probably get everything running using
virtualenv and setuptools, but at that point I'm refactoring the entire
package structure.  Breaking backwards compatibility by eliminating
dependencies would help but not solve the problem entirely.

At this point I'm stopping for now, primarily because I've changed the
objective of the project I was planning on using PEAK-Rules for so it's
no longer useful.  I didn't tackle any of the hard problems like
rewriting ByteCodeAssembler.  My attempt is up at
https://github.com/ceridwen/PEAK-Rules  If anyone
needs a Python 3-compatible version of Extremes or SymbolType, or a
mostly-compatible version of DecoratorTools, there they are.

Cara