[PEAK] PWT: layout, macros, parameters, etc.

[Phillip J. Eby]

I've now finished most everything I planned for peak.web this month, except 
for the sitemap parser (the raw sitemap functionality is implemented, just 
not the parser to activate the functionality from XML) and the advanced 
facilities of the next-generation PWT language.

So, this post is to help me flesh out those "next generation" features, 
which up to this point have only been sketched.

The main thing I want to touch on is the "layout" or "macro" facility, that 
should allow us to "call" another template or DOMlet with blocks and data 
designated as named parameters.

After giving it some thought, I don't think we need a separate ':layout' 
attribute to invoke macros or layouts.  Instead, the standard ':replace' 
can be used.  Normally, 'replace' expects to convert the target data to 
unicode, but we can first adapt to IDOMletNode.  If the target to be 
rendered is a DOMlet node, we simply invoke it in-place (after setting up 
any parameters).

This would mean that defining a view on an object to map to a template or 
DOMlet, would automatically cause it to be rendered inside a referencing 
document, and changing a view from a Python string expression to a template 
or back would not affect templates using the view.

So how do the parameters get passed?  I'm envisioning the creation of a 
context object, pointing to a mapping that contains the parameters.  This 
context would be passed to the invoked DOMlet as the current context, so 
any path usage in the invoked DOMlet would see the parameters.

That concept works nicely for a DOMlet being used as a macro, but not so 
well for a view on an object.  A view on an object really wants to have 
only one parameter: the object to be viewed.  And, it should bind tightly 
to the object being viewed; it shouldn't be possible to refer to the view 
and then somehow disconnect it from its target.

So, when we register "resource" and "object" views, we need to include some 
glue code that asks the view object if it would like to "bind" to the 
object being viewed, by returning a "version of itself" that's linked to 
the target.  In the case of a DOMlet-based view, this would allow it to 
ignore any extraneous parameters.  (We don't need this code for "attribute" 
or "function" views, since presumably these can do their own binding.)

Of course, I suppose it's possible to have a situation where you need to 
pass parameters into a view on an object, but I don't see a clean way to 
support that, without forcing some sort of prefix convention like 
'here/foo' on every view template (the way I think ZPT does).  I suppose it 
might not be so bad, if the prefix were '..', such that '../foo' referred 
to 'foo' on the view's target object, and 'bar' referred to the 'bar' 
parameter.

But that seems awfully backwards.  Maybe we could add a 'params' attribute 
to ITraversalContext instead.  Then, the parameters could stay with the 
traversal no matter what, and they can be accessed with 
'/params/whatever'.  I don't think this generalizes well, though.  For 
example, I wouldn't want to use '/params' as the way to access sequence 
variables (like DTML's "sequence-index") inside a "list".

But it would work a lot better if the data were actually part of the 
IDOMletState, because parameters are part of the execution state, not the 
traversal context.  After all, you can traverse to '..', which would take 
you to a context where the parameters don't apply any more.  That's just 
too wonky.

So, here's what we'll do instead.  The IDOMletState will hold a mapping for 
parameters, variables, and whathaveyou, that will get cloned whenever you 
create a new DOMletState.  For example, if we're passing parameters to an 
IDOMletNode view, we'll clone the current state and put the parameters in 
under a 'params' key.  Then, whenever we do path traversals  that begin 
with a '/' -- which currently traverse to the traversal context object -- 
we'll instead traverse to a wrapper that layers the DOMletState's mapping 
on top of the traversal context.  Thus, both context-specific and 
state-specific items will be available to templates.

So /params/foo will get you to the foo parameter, /user gets you the 
current user, and so on.  If the 'list' domlet grows sequence variables 
like DTML's sequence-index, you'll be able to use paths like 
/sequence/index to get at them.

Excellent!  There's only one other open issue for parameter passing, and 
that's multiple parameters with the same name.  Currently, DOMlets allow 
you to specify more than one parameter with the same name, which is useful 
for e.g. the 'list' DOMlet to rotate through alternative formats of list 
items.  Anyway, I don't think this approach can continue, because templates 
using these things as parameters won't have any way to tell if a given 
parameter is a single value, or a list.  So, I think we need to go 
"strongly typed" here, and either banish multi-valued parameters, or have a 
different way to specify a single value versus a list value.

So, let's say we used 'this:is' (and 'content:is') to mean a single-valued 
parameter, and 'this:is-a' to mean a multi-valued parameter?  Then, even if 
you only supply one 'is-a', the resulting parameter is multi-valued (i.e., 
a list).

I'm not sure this is the best thing either, though.  So far, there's only 
one DOMlet we use that takes even one multi-valued parameter.  And, more 
often than not, you're only giving it one value: the format for all list 
items.  So, it's going to be weird trying to remember when to use ':is' 
versus ':is-a'.

Maybe the solution there is to use a different parameter name for rotating 
list items, e.g. 'this:is-a="rotatingFormat"' when you need rotating 
formats, and use 'this:is="anItem"' normally.  Hm.  This also shows that 
parameter names can suggest by their spelling, what mode is used to invoke 
them.  So we could say 'this:is="theHeader"' and 'this:is="theFooter"' to 
indicate list header and footer, although that seems a wee bit over the top.

Really, I'm having a hard time coming up with use cases for multi-valued 
parameters besides ones that are passed to Python-based DOMlets, as there 
are lots of other ways to do things in templates.  Especially since 
multi-valued parameters specified in a template are always going to be 
fixed, functionlike values.

Maybe it would be better to just drop the whole idea, and make the 
'list'  DOMlet work harder to get rotating formats.  E.g. make it check for 
parameters named 'format-1', 'format-2', and so on.  That would be pretty 
damn ugly, but at least it could be processed at "compile time", so to 
speak.  Maybe I could just allow DOMlets to specify multi-valued argument 
names at "compile time", and disallow all others.  This would mean that 
DOMlets written in Python (like 'list') could use multi-valued parameters, 
but dynamic template-based DOMlets chosen at runtime would always receive 
single-valued parameters.  That seems like a reasonable compromise.

Okay, I think we've got the invocation and parameter model sorted 
out.  What's left?  Consulting my to-do list, I see:

* dynamic attribute values
* path alternatives
* /nothing, /default, and /modules
* :if/:unless
* :try/:except
* :expects

So let's go through them.  Dynamic attribute values.  It would be nice to 
be able to specify image width and height attributes.  I had been thinking 
about stealing ZPT syntax, which is something like 'tal:attributes="width 
widthexpr; height heightexpr"'.  But today I saw that Nevow has something like:

<img src="something"><nevow:attr name="width"><nevow:slot name="widthval" 
/></nevow:attr><nevow:attr name="height"><nevow:slot name="heightval" 
/></nevow:attr></img>

Which is actually pretty sucky, compared to the ZPT syntax.  But, it leads 
to this idea:

<img src="dummy"><span this:attr="width" this:replace="widthexpr"/><span 
this:attr="height" this:replace="heightexpr"/></img>

...which is somewhat more interesting.  As long as you don't include any 
whitespace between the tags, the '<img>' tag as rendered would be an empty 
tag.  The interesting question, of course, is whether other tools would 
barf on the non-empty '<img>' tag in the source.

Another possibility is just to have an attribute namespace, like this:

<img src="dummy" attr:width="widthexpr" attr:height="heightexpr">

This would work for maybe 90% of the use cases; in the rare scenario where 
you need to also specify an XML namespace, you could maybe do:

<something ns_for:foo="bar" attr:foo="foo_expr">

Which at runtime would set a 'bar:foo' attribute to the value of "foo_expr".

Yeah, I think I like this.  In order to make it work, we're going to have 
to have an interface to let you tell DOMlets about dynamic 
attributes.  Which we could refactor the existing 'url' DOMlet to 
use.  Interestingly, once we had that interface, it could also be used for 
the more Nevow-like approach of using nested DOMlets to tell an outer 
DOMlet about attributes.  But I don't see any immediate use for that.

Excellent.  Let's move on to path alternatives, /nothing and /default.  I 
think we're often going to run into the case that a value we're looking for 
doesn't exist, or even more often, the user isn't allowed to get to.  ZPT 
handles this by allowing alternative paths, e.g "foo|bar" to mean, "look 
for foo, and if you can't get it, try bar next".  Often "bar" is actually 
"nothing" or "default".  "nothing" means, "don't put anything here", and 
"default" means, "put whatever is in the template here".  These are both 
common use cases.

We could allow path alternatives, and add 'nothing' and 'default' objects 
to the context namespace, such that "foo|/nothing" and "bar|/default" would 
do what we want.  But this looks ugly, and if these are common cases, it 
would be nice to have a more compact, yet more readable syntax.

There are three states we want to distinguish in the common cases:

* If this thing isn't accessible, blow up with an error
* If this thing isn't accessible, don't display anything
* If this thing isn't accessible, display something else instead

There are some interesting distinctions between these cases.  For example, 
I think that the "don't display anything" case will usually apply to a 
bigger block than the one where the value is being looked for.  Consider an 
output form, where you are displaying various fields, perhaps using a 
table.  You'll want to omit the entire table row where an inaccessible 
field occurs, not just the cell where you're outputting the value, e.g.:

   <tr>
     <td>Field X:</td>
     <td content:replace="field_x">Value here</td>
   </tr>

What you really want is to do something like:

   <tr this:if="field_x">
     <td>Field X:</td>
     <td content:replace=".">Value here</td>
   </tr>

That leaves the question of alternative/default output.  For attributes, I 
think we can use the ZPT style of alternative paths, 
e.g.  'attr:foo="something|/nothing"', or 'attr:bar="foo|/default".  For 
attributes, this notation is reasonably compact, and I don't expect 
attribute mangling to be common anyway.  'nothing' would mean, "don't 
include the attribute at all", and 'default' would mean, "use the 
attribute's existing value in the template".

For default output at the element level, I'm a bit more torn.  For example, 
consider the 'list' DOMlet.  Usually, you're going to have some kind of 
structure, let's say:

    <ul content:list="foo">
      <li this:is="listItem" content:replace="title">Item here</li>
    </ul>

But, if "foo" is empty or unavailable, you probably don't want the '<ul>' 
to even appear, e.g.:

    <ul this:if="foo" content:list=".">
      <li this:is="listItem" content:replace="title">Item here</li>
    </ul>

And maybe you want a <p> to appear instead, saying that there are no items 
available:

    <p this:unless="foo">No foo items available.</p>

This is becoming altogether too much like programming.  Also, the ':if' 
doesn't look as though it should traverse to the thing it points at.  It 
almost seems that ':with' would make more sense:

    <ul this:with="foo" content:list=".">
      <li this:is="listItem" content:replace="title">Item here</li>
    </ul>
    <p this:unless="foo">No foo items available.</p>

So, the idea could be that 'with' traverses to the item, but if the item 
does't exist, the whole block is skipped.  In other words, "execute this 
block 'with' this object, if it's available."  I'm not sure whether 
'unless' also needs renaming.  It clearly can't change what the current 
object is, since its content only executes if the target doesn't exist.

However, this brings up a different issue.  Should 'with' and 'unless' pay 
attention to the boolean value of their subjects, or just their 
accessibility?  What about the '|' operator?  Should it pay attention?  So 
far, the above use case is the only one that suggests there's value in 
paying attention to it, and maybe it could be handled differently:

    <ul this:try="" content:list="foo">
      <li this:is="listItem" content:replace="title">Item here</li>
      <p this:is="emptyMessage" this:catch="NotFound,NotAllowed">No foo 
items available.</p>
    </ul>

The idea here is that the outer block is a 'try' block; it tries to execute 
its contents, and if there's an error, it checks its parameters for a 
matching 'catch' block, and then replaces itself with that 
block.  Meanwhile, in the example above, that block is also being used as 
an 'emptyMessage' parameter to the 'list' DOMlet, so the same message is 
used if the list is empty.

Hm.  It works, but it's now horribly ugly when compared to its 
predecessor.  Of course, we *could* spell it like so:

    <ul this:with="foo" content:list=".">
      <li this:is="listItem" content:replace="title">Item here</li>
      <p this:is="emptyMessage">No foo items available.</p>
    </ul>
    <p this:unless="foo">No foo items available.</p>

at the cost of some duplication.  But I'm guessing that in the common case, 
it's more likely that you'll be doing:

    <ul this:with="foo" content:list=".">
      <li this:is="listItem" content:replace="title">Item here</li>
      <p this:is="emptyMessage">No foo items available.</p>
    </ul>

and just not saying anything if the user isn't allowed to access 
'foo'.  No  sense making a big deal out of it.  Anyway, I think we'll want 
to have 'with' and 'unless' ignore the boolean value, and deal only with 
accessibility.  There are plenty of other ways and places to deal with the 
boolean nature, such as through views, and Python DOMlets.

Finally, I think we'll have 'default' map internally to NOT_GIVEN, and 
'nothing' map to NOT_FOUND.  That will probably make the most sense.

Okay, so what's left now?

* /modules
* :try/:catch
* :expects

The '/modules' thing is an idea stolen from ZPT, which has a 'modules' 
mapping to access arbitrary Python modules.  We'll put it in the 
DOMletState variables mapping, so it'll only be accessible from within 
hardcoded template paths.  Technically, it won't give access to the 
modules, but rather to their dictionaries.  So, 
'/modules/foo.bar/something' will look up 'something' in the 'foo.bar' 
module's dictionary.  This is because our security machinery considers 
dictionaries' contents to be "fair game" for anybody to look at, unless the 
object pulled out of the dictionary forbids access to itself.  But, it 
doesn't allow access to attributes unless they are explicitly 
granted.  Anyway, the purpose of this is to allow you to pass Python 
objects as parameters to DOMlets or templates.

Of course, if the object is a function or class or something like that, you 
won't be able to actually *do* anything with it from a template, unless it 
can be adapted to an IDOMletNode, or it's something like a string, or if 
you have some kind of view declared on objects of that type.

Okay, what about try/catch?  Well, the main use case I had in mind for them 
was to deal with inaccessibility of locations, and that's going to mostly 
be handled by with/unless.  And I worry that having them will increase the 
temptation to use PWT as a "scripting" language, instead of merely as a 
templating system.  So, I think I'll call YAGNI on them for the moment.

':expects', on the other hand, is just an assertion as to the type or 
protocol of the thing currently being handled.  E.g., 
'this:expects="mypkg.IFoo"', to indicate that the current object should be 
adapted to 'IFoo' before proceeding.  More commonly, I imagine you'll do 
'this:with="foo" content:expects="mypkg.IFoo"'.  Since this is an 
essentially static declaration, the string should be an import string 
that's executed at compile time.  It's hard to think of a meaningful use 
case for this with a dynamic target protocol, so I'll call YAGNI on 
anything but statically defined protocols.  It's probably creeping 
scriptism, that should be moved into a Python DOMlet specialized to the task.

Hm.  Rechecking my offline to-do list, I've still got a few items left to 
sort out:

* with:paramName="path"
* page:layout="layout_path"
* page:content-type="something/something"

The first one is a syntax for passing path parameters to a template or 
DOMlet.  There are two open issues with it.

First, should the path be interpreted eagerly (before calling) or lazily 
(by the called template/DOMlet)?  There are arguments both ways.  Lazy 
means that bugs could be harder to track, if the value is maybe passed down 
multiple calling levels.  On the other hand, eager means that you can't 
conditionally not bother using one of the parameters.  I think I'm going to 
lean towards being lazy here, since it enables more things.  Maybe there's 
a way to have the error diagnostic report the location at which the path 
was specified, not just the location at which the problem was detected.

So, on to the second issue: what DOMlet do the parameters get sent 
to?  This could be tricky if there are two DOMlets sharing an 
element.  (That is, if there is both a 'this:' DOMlet and a 'content:' 
DOMlet on the same element.)  And, if there's neither, do the parameters 
"bubble up" to the next outer element?

I guess my inclination is to say that the parameters go to the innermost 
element that wants parameters.  For example, in:

   <ul this:with="foo" content:list="."
       with:listItem="++resources++/foo.bar/someTemplate">
     <li>This is just sample data that will be ignored.</li>
   </ul>

We want the 'listItem' parameter to be a specified template resource, 
passed to 'content:list'.  But here:

   <span this:replace="@@someview" with:mode="simple">
   blah blah
   </span>

we want the 'mode' parameter to go to the '@@someview' being invoked.

I don't actually have a use case for these parameters "bubbling up" to a 
containing element at the moment, except in cases where the parameter needs 
to be multi-valued (and the target element allows it at compile 
time).  However, the current parameter machinery that already exists, does 
this kind of bubbling up, so in the absence of any reason *not* to do it, 
we'll just use that existing machinery.  Currently, parameters are required 
to implement IDOMletNode, so we may have to do some things to support these 
new lazy path expression parameters.

Okay, so that's all settled now.  On to the 'page:' attributes.  In this post:

http://www.eby-sarna.com/pipermail/peak/2004-August/001715.html

I suggested using 'this:page_layout' to specify a DOMlet that would be used 
to wrap the contents if the template were accessed as a web page, as 
opposed to being used as a fragment in another template.  If the value were 
empty or "None", it would mean the page was standalone, and the default, if 
no layout was specified, would be that the template could *only* be used as 
a fragment.

But, if we have 'with:' and ':is' available, I don't think we need these 
any more.  Instead, we can define *parameters*, that bubble up to the 
TemplateDocument object that is the template's top-level element!  For 
example, sticking 'with:layout="/default"' on the 'html' element of the 
document could mean, "this is a standalone template", versus 
'with:layout="some/path_expr"' to select a specific layout.  Or, maybe 
'this:is="fragment"', to mark the portion of the document that should be 
used as a fragment, and 'this:is="document"' to mark a portion to be used 
as a web page.

Hm.  Let's consider the use cases.  We want to be able to:

* Take a complete document and slice out a portion of it to be used as a 
fragment, when invoked from another template

* Take a fragment, and embed it in a dynamically chosen layout.

* Deal intelligently with data that's outside of any XML/XHTML element, 
such as the '<?xml' preamble, '<!DOCTYPE', and DTD stuff.  ("Intelligently" 
would mean, being able to consider it part of the document, if it is.)

Hm.  So how about these special parameters:

* with:page-layout="some_layout"  (or 'this:is="page-layout"')
* with:fragment-layout="something_else" (or 'this:is="fragment-layout"')
* with:content-type="text/whatever"

then, the document will either invoke page-layout or fragment-layout, 
according to invocation mode.  In either case, it will pass in any other 
document-level parameters.  So, if you have a document that defines 
parameter blocks (this:is or content:is) to refer to a page header, 
sidebar, footer, and so forth for the layout to include, then these will be 
passed to either the fragment layout or the page layout.  And, the fragment 
layout could ignore everything but the body content, while the page layout 
would include all that stuff.

One thing you could *not* do, however, is use this/content parameters to 
specify a fragment nested within a document.  In other words, you can't 
take the document as a whole and then say, hey, use this bit inside here as 
the fragment, and this outer part as the document.  At least, you can't if 
there are any parameter-accepting DOMlets between the two, because that 
will stop the "bubbling-up" process.  In principle, you can do it; it's the 
practice that could be tricky.  But I guess if you aren't manually invoking 
a layout component on the document or any subset thereof, then this would 
work just fine.

So, now the semantics: if you don't specify a page layout, or use /nothing, 
the template shouldn't be usable as a page.  If you use /default, then the 
entire document, preambles and all, is the page layout.  If you don't 
specify a fragment layout, or you use /default, then the whole template is 
the fragment layout.  If you use /nothing, then the template isn't usable 
as a fragment.  So, the semantics for explicit values are the same, but 
omitting the parameters has different meaning, defaulting to the safer 
"don't allow access from the web" mode.

I'm almost tempted to say the parameter names should just be 'page' (or 
'document') and 'fragment'.  These make more sense when used with 'this:is' 
and 'content:is'.  But, when used with 'with:', the '-layout' names make 
more sense.  Maybe it should just be that way.  That is, you use '-layout' 
names to designate something that will be called with the other parameters, 
and the non-layout names to designate a block to be used.  So, 
'with:page-layout="something"', versus 'this:is="page"', and similarly for 
fragment-layout and fragment.

Whew.  It's been a long haul, but worth it.  I think we now have a syntax 
and execution model that offers a great deal of flexibility, while still 
mostly avoiding the pitfalls of turning into a scripting language.  Its 
main syntactual drawback is the need to define a lot of XML namespaces:

* this
* content
* with
* attr
* ns_attr

But probably 80% of documents can get by with just the first three, and 
maybe 99% or more can get by with the first four.  And, you don't have to 
define any of them if your template isn't being run through tools that care 
about valid XML namespaces.  I'm also thinking it probably wouldn't be hard 
to write a tool that would add the namespace definitions for you, pulling 
the data from PEAK's default namespace settings.

Finally, I think I'm going to treat the /modules thing as a YAGNI for 
now.  The only thing left that might have used it is ':expects', but that's 
going to use an import string directly.  If you need constants and the 
like, you can easily define them as views on relevant objects from your 
sitemap, and sitemaps allow easy importing.  I think we'll need to wait for 
more "field experience" with these matters to see if the shortcut is really 
useful.

One final question: /params or /args?  Which name makes more sense as the 
place where you find the things your template has been called 
with?  /params feels a bit more "right" to me, but /args is much shorter to 
type while still conveying the same basic idea.

Thoughts, anyone?