[PEAK] PEAK Roadmap Update

[Phillip J. Eby]

I've finally started using a bit of my spare time towards PEAK development 
again.  If you're subscribed to the source-changes list, you'll have seen a 
bunch of commits lately splitting out peak.util.imports to a new 
"Importing" project, as well as a new "BytecodeAssembler" distribution 
split from RuleDispatch.

Next up is splitting out PyProtocols' class and function decorator features 
into a new "Decorating" project, where they'll be reusable by other packages.

After that, I'll be starting on a new PEAK-Rules project, whose purpose 
will ultimately be to replace both RuleDispatch and PyProtocols, although 
it won't do so immediately.

PEAK-Rules will basically be a ground-up rewrite of RuleDispatch based 
entirely on generic functions, with no interfaces or PEP 246 
adaptation.  This is a reflection of the fact that both Alex Martelli (PEP 
246's primary champion) and Guido have both caught on to the greater power 
of generic functions.  PEPs 245 and 246 (interface syntax and adaptation, 
respectively) have been rejected, and Guido has a simple generic function 
prototype he has implemented for Python 3.0 (coming in a couple of 
years).  You can see Guido's prototype at:

http://svn.python.org/projects/sandbox/trunk/overload/

I created an enhanced version of Guido's prototype that actually uses a 
self-bootstrapping set of generic functions to implement generic 
functions.  :)  It is designed to be supremely extensible, such that a 
RuleDispatch-like system can be implemented atop it.  The source of my 
prototype is here:

http://svn.python.org/projects/sandbox/trunk/Overload3K/

After doing these, it became clear to me that rather than try to refactor 
RuleDispatch as it is currently built (which was my original plan for 
future enhancements), I would be better off rebuilding it from scratch atop 
a microkernel like this one.  In that way, it will be entirely independent 
of PyProtocols, and I will be able to correct its many design flaws in the 
process.

Specifically, the interface/adaptation/concrete methods approach to 
building RuleDispatch forced a lot of design and implementation compromises 
that I now see are unnecessary if you have generic functions 
available.  For example, there are a lot of kludgy implementations of 
__and__ and __or__ and other binary operations, that would be cleaner if 
they were implemented via generic functions.  Also, the entire method 
combination approach (when/around/before/after/etc.) is terribly kludgy, 
and I have figured out how it can be replaced with a more easily extensible 
system.

Last, and far from least, the dispatch tree building and code generation 
machinery is too hardwired in RuleDispatch; there is no easy way to vary 
that mechanism independent of everything else.  If you want to modify 
anything about a generic function you have to subclass it, whether you're 
just changing the combination rules or you're changing the tree building or 
indexing.

So, PEAK-Rules will implement these using separate components: Rules, 
RuleEngines, and RuleSets.  RuleSets will hold rules and will be able to 
influence how method combination works.  RuleSets will also be able to hold 
configuration "hints" that can influence how indexing and code generation 
will work.  But the actual indexing and code generation will be controlled 
by RuleEngine objects, which will mostly be invisible to the framework 
user.  Only if you need to create custom implementation strategies will you 
need to work with RuleEngines.

I'm hoping that PEAK-Rules will have a much smaller API footprint than 
RuleDispatch+PyProtocols.  It's possible that the total implementation size 
will be smaller as well, although it's way too soon to tell; I haven't even 
created the project directory yet, just done some design sketches to work 
out how the basic features will work.

I'm really excited about the design, though, because it's *much* more 
orthogonal than the current super-monolithic RuleDispatch 
implementation.  It should be possible to experiment with and test a whole 
lot of stuff without actually introducing the full Python expression and 
indexing support, using just the microkernel features.  And the fact that 
code generation strategies will be equally pluggable means that I'll be 
able to trade off some complexity for flexibility.  That is, rather than 
trying to have the One True Dispatch Algorithm (as RuleDispatch does) that 
wants to be more-or-less-optimal for every possible situation, I'd like to 
make PEAK-Rules able to be tuned for good performance in typical cases, 
with the ability to give "hints" to adjust the chosen algorithms for 
less-typical cases.

Anyway, at this point RuleDispatch and PyProtocols are pretty much 
feature-frozen.  I will still fix bugs, of course, but there aren't going 
to be any new features; such efforts will go into the new package.  It's 
not clear to me yet if PEAK-Rules will offer an Interface type or other 
PyProtocols-like APIs.  To some extent it depends on where the rest of PEAK 
is going.

At some point, peak.events is going to split out into a separate project, 
probably called "Eventual".  It'll have some changes to help it work better 
with Python 2.5.  It'll probably also require the "Contextual" library 
(which is already in SVN).  "Contextual" is a library for dynamic context 
management that will offer an attractive replacement for PEAK's existing 
configuration system.  There may be some interaction between PEAK-Rules and 
Contextual at some point, though I'm not sure how that's going to work yet.

At some point in the indefinite future after all that, Schematic (aka 
peak.schema) will raise its head, as it will depend on at least Contextual 
and PEAK-Rules, and whatever the naming, binding, and storage packages will 
have morphed into by then.  So that stuff should all be ready by the time 
Python 3000 comes out.  :)

So there you are, my roadmap update.  No dates on anything, since this is 
all spare-time stuff for me unless somebody wants to provide funding (hint, 
hint).  ;)  However, I anticipate being able to have a demo-able version of 
PEAK-Rules in June.  It may not have all of the features of RuleDispatch at 
that point, but it should be able to do other things that RuleDispatch 
can't (and won't ever) do.  The "Decorating" (DecoratorTools?) project 
should be out this week or next, as it needs to be done before I can really 
start PEAK-Rules.