[PEAK] Reactor-driven microthreads

[Bob Ippolito]

On Dec 31, 2003, at 1:12 PM, Phillip J. Eby wrote:

> At 12:12 PM 12/31/03 -0500, Bob Ippolito wrote:
>> On Dec 31, 2003, at 12:22 AM, Phillip J. Eby wrote:
>>
>>> Hm.  This seems like the first really good use case I've seen for
>>> having a macro facility in Python, since it would allow us to spell
>>> the yield+errorcheck combination with less boilerplate.  Ah well.
>>
>> That, and "flattening" generators
>>
>> def someGenerator():
>>     for something in someOtherGenerator():
>>         yield something
>>
>> This gets really ugly after a while, but it happens a lot when you are
>> doing "microthreads".
>
> Actually, the Thread class in my second post allows you to do:
>
> def someGenerator():
>     yield someOtherGenerator(); resume()
>
> Thread objects maintain an external "stack" of nested generators, so 
> you don't need to do this kind of passthru.

That is the other way to do it, but I don't like that at all.  It's not 
a general solution, I use this 'pattern' very often with or without a 
'scheduler' on the outside.

>> By the way, the microthread thing in Twisted has been talked about
>> before (
>> http://twistedmatrix.com/pipermail/twisted-python/2003-February/ 
>> 002808.html ) and is implemented to some extent as twisted.flow
>> (entirely different from the flow module in that discussion).
>
> I've looked at twisted.flow.  To me, it's way overcomplicated in both 
> implementation and concepts.  By contrast, the approach I've just 
> sketched involves only two interfaces/concepts: threads and 
> schedulers.  (Three, if you count "reactor".)  By contrast, 
> twisted.flow seems to have stages, wrappers, blocks, and controllers, 
> just for starters.  (Again, not counting "reactor" as a concept, and 
> ignoring that the package also depends on Deferreds.)  Last, but not 
> least, twisted.flow doesn't support the kind of error-passback 
> handling that I've devised, unless I'm misunderstanding something 
> about how it works.

I didn't write and I don't use twisted.flow so I have no idea, but I 
think that the code that I posted in the mailing list is somewhat 
roughly equivalent (in a same-concept-but-not-nearly-as-polished kind 
of way) to yours though... except it requires you to yield a particular 
kind of object from your iterable "microthread" and that object has 
your "resume" method.  The only real thing about your implementation 
that I disagree with is this global resume function.  Oh, and I 
disagree with the fact that you say "generators provide an ideal 
basis".  Ideally, you'd be using Stackless Python.

In any case, I use code like this quite often with and without Twisted 
and agree that it helps in quite a few situations but in others you 
really ought to be using an event driven model (when what you're doing 
isn't nearly as linear, which does happen).  A facility like yours will 
surely lower the barrier to Twisted programming because as you said 
most programmers are far better linear/iterative thinkers.  The real 
problem is, of course, that Python sucks at this concurrency which is 
why Twisted feels like the tail wagging the dog.  I think that things 
could be made a LOT nicer with macros, but I don't see a version of 
Python with macros happening anytime soon.

-bob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2357 bytes
Desc: not available
Url : http://www.eby-sarna.com/pipermail/peak/attachments/20031231/3bff15a0/smime.bin