[PEAK] DecoratorTools, Aspects, and the Trellis

[Phillip J. Eby]

So at this point I have a rather splendidly-working prototype of the 
new Trellis update algorithm, sitting at around 200 lines of code in 
the svn.eby-sarna.com repository.  However, I don't trust it has 100% 
code coverage, because I wrote it basically from scratch, ran it 
against an older test suite, and then added some tests to exercise 
some of the new functionality.

In the process of doing this, I found that there are lots of ways in 
which you could do things that would foul up the implementation, that 
were not covered by tests.  In other words, I found potential bugs by 
inspection, that needed their own proofs of impossibility.  I've 
added tests for the things I've thought of so far, and fixed the bugs 
as I found them, but I suspect there may be one or two more lurking 
beneath the surface.

So I'm going to do one more draft of the code, but this time using 
TDD, both to be absolutely sure I know how the thing works, and also 
to possibly improve the efficiency a bit, although I expect the 
algorithm in its simplest form to be a straightforward translation to 
C if needed.  I've already done some memory-related optimization, 
such that Trellis' Cell objects consume less memory than their 
PyCells counterparts, and about 1/4th the memory consumed by the 
competing "Cellulose" library.

I expect that even a modestly-sized Trellis-based program, using 
cells to simultaneously manage its GUI, persistence, and network I/O 
will be capable of having thousands or even tens of thousands of 
cells in use at any one time, so both space and speed efficiency are 
important for a production system.

The other big step that remains to be done on the core Trellis API is 
to create the class and function decorators I've sketched here 
previously.  In the course of planning how to implement them, it's 
occurred to me that there are some recurring patterns in how I've 
done these sorts of decorators for Chandler, Contextual, and other 
projects, and that I can and should add code for these patterns to 
the DecoratorTools package, so I don't have to keep recoding them in 
an ad-hoc way.

The PEAK-Rules package contains an "Aspect" implementation that's 
used to dynamically attach additional information to existing 
objects, sort of like you might slap a sticker on the side of an 
object in real life.  The advantage of using an Aspect over poking 
extra attributes onto the object is twofold: 1. you don't have to 
come up with a name that doesn't collide with anything else, and 2. 
you don't have to write code to *add* the aspect to the object.  Just 
invoking the aspect constructor is enough to either return the 
existing aspect, or create one and attach it somewhere.

And it has occurred to me recently that this bit really belongs in 
DecoratorTools rather than PEAK-Rules, because with it, you can write 
method decorators that can effectively add "extra attributes" to the 
class' instances.  Like, oh, say...  a map of attribute names to 
trellis.Cell objects.  :)

The second thing that occurred to me, just today, is that the 
decoration of the *class itself* could be done using aspects -- 
aspects attached to the class, rather than the instance.  For 
example, in the Trellis case, we also need information about the 
rules and default values of the cell attributes.

Unfortunately, regular class decoration as currently implemented in 
DecoratorTools is pretty much limited to sticking attributes on the 
class or doing things to it after it's already created -- which is 
much more limited in what can be done with the class.  For example, 
it's no use trying to change __slots__ after the class is created, 
nor can you change a class' metaclass after creation, unless it was 
created using a metaclass other than 'type'.

Granted, for a lot of uses it's not a terrible tragedy.  In my 
examples, I've been using base classes like trellis.Component to 
indicate that some sort of mixin is required, in order for the rules 
to work.  The trickier bit is ensuring that people don't use 
decorators like trellis.rules(), and expect them to work on a 
non-Component class.

My current solution for things like Contextual's "context.replaces()" 
decorator is to add a class decorator that checks the created class 
for suitability.  This ensures that you'll at least get an error, and 
won't be able to proceed without either getting rid of the decorator, 
or adding the right base class.  It could be nice, however, if you 
could just have the decorator add in the mixin.

On the other hand, I can think of some folks who could complain that 
not including an explicit base or metaclass is likely to make code 
harder to debug or understand, especially if someone's not familiar 
with *all* the decorators in a body of code.

So we can probably live with that for the Trellis.  The 
trellis.Component mixin isn't in fact going to have a lot of stuff 
added to the class anyway; just a default __init__ method to handle 
keyword arguments, and a metaclass __call__ implementation that will 
make sure that any eager cell attributes are activated once __init__ 
has completely finished running.

So, I'm thinking that the idiom I have now of adding class decorators 
to check this, could be simplified a little by adding a class 
decoration function like ``class_requires(message, func, *args)``, 
that raises TypeError(message) unless func(cls, *args) returns 
true.  I could then use that in place of the custom code in 
Contextual, and Trellis needs it too.

The next pattern I find myself repeating is the need to process 
inherited metadata from a class' base classes, either by base order 
or MRO order (or the reverse), in order to figure out the combined 
metadata for the new class.  This usually involves a lot of private 
attributes rudely shoved into the class dictionaries, and a loop that 
peers into the base classes, looking for whether those attributes are there.

It would be a lot nicer to write code like:

     for base_aspect in CellMetadata.by_mro(cls):
         # combine some sort of metadata from the base_aspect object

Of course, there are some non-trivial problems to be solved 
here.  Unlike attributes, aspects can't be inherited (unless I make 
them able to be).

Second, at one of the times when we most want to use these class 
aspects, the class itself (including its bases, mro, etc.) doesn't 
exist yet!  So there has to be a way to access and set up the aspect 
before the class exists, then have it do any needed setup later.

However, the solution to the second problem also points to a way of 
solving the first.  If class aspects can be set up lazily, then we 
don't have to worry what happens with base classes not having an 
aspect, because we can simply create them as we go -- even for 
classes that never had the aspect to start with!

(For example, the 'object' class can't be decorated in any normal 
way, but an aspect can certainly be attached to it afterwards.)

In order to implement this, class aspects will have to be looked up 
in a weakref-keyed dictionary, if they're not already in the class' 
dictionary.  (Because new-style class dictionaries can't be directly 
modified once the class exists.)  But, for pre-creation aspect 
access, one can create the aspect inside the class dictionary before 
the class exists, then add a class decorator to invoke the 
after-creation setup.

So, class aspects could have a method like 'setup_for_class(self, 
cls)', that would get called to allow the aspect to initialize 
itself, using any data it wants to from the class object.  It can 
also request aspects of the class' bases or mro, and those aspects 
can then lazily come into being -- and have *their* setup_for_class() 
methods called, if need be.

So, instead of the pattern I use now, where I have to loop over base 
class info and fill in the metadata gaps for classes without it 
(which can be tricky at times), I can instead just let every base 
have an aspect with appropriate registry attributes, each of which 
knows just how to assemble itself from its nearby bases, or its whole 
mro if need be.

Yep, that sounds about right.  Boy I wish I'd had this idea of 
aspects, and especially class aspects, when I was writing the PEAK 
core a zillion years ago when Python *2.2* was the latest and 
greatest.  I would never have created all those special attributes 
like __objectsToBeAssembled__ and __class_offers__ and so on.  But 
then, if I'd had Contextual and the Trellis back then, I wouldn't 
have needed to write a good chunk of the PEAK core in the first place.  :)

So, once I have aspects moved from PEAK-Rules to DecoratorTools, and 
add these new "class aspect" things, I'll be able to start on the 
Trellis class-level API, so you can declare the rules and default 
values and whatnot.

I'll start on that tomorrow.