[PEAK] Re: Trellis-fork

[Sergey Schetinin]

Next thing I want to do it introduce thread safety. There's some code
to help do that in Trellis already, but I plan to remove that
implement a very simple model. If I am not mistaken, Phillip planned
for cells to have locks in their manager attribute, so that
ctrl.lock(cell) uses that just like other managers, calling __enter__
/ __exit__. For locks that acquires and releases them so, given
ctrl.lock is called in all appropriate places the access to cells is
synchronous.

The problem with this is that if different cells have different locks
as their managers the deadlocks are very likely. There's no way of
telling what cells will be used in a transaction and in what order so
two concurrent transactions are very likely to get into a deadlock.

If all cells have the same lock as their manager there are no
deadlocks, but the whole ctrl.lock() business is pointless as there's
only one lock to acquire and release.

Another possible case is when there are two or more sets of cells that
form disjoint networks. All cells in each of these sets would have the
same lock. This way there's a guarantee of no deadlocks, but the only
reason to use locks in this case is if there will be concurrent
transactions on the same cell network.

I think the best tradeoff for now is a GIL-like solution -- just
disable all this cell.manager / ctrl.lock stuff and just hold the
global lock in ctrl._process. This doesn't support the last case
described above, but let's first have an actual use case for that and
see how it could be supported then. One simple solution that could
work for that is to have a controller per cell group / network with
its own lock.

Another problem is how cells store their values -- they change them
in-place and log that for undo. The problem is that if that cell is
accessed from another thread, and from *outside* a transaction the
changed value will be seen -- that's not good at all. For example see
_ReadValue.get_value implementation:

    def get_value(self):
        if ctrl.active:
            # if atomic, make sure we're locked and consistent
            used(self)
        return self._value

If not atomic there's no guarantee of getting valid value which isn't
easily resolved.

One way to fix that is to store changes in a different attribute and
read that only if inside transaction. A development of this approach
is to let the controller store all of cell values. This also makes way
for properly concurrent transactions without locking because the
controller can store the consistent data snapshot in one location and
let the transactions work on a copy. Each transaction would get its
own copy so there's no need to lock at this point. At commit time
however the changes need to be merged, and because all of value
storage is handled by controller it can verify at this point if other
transactions changed any inputs used by this transaction since it
started. (For more on this see
http://code.google.com/p/trellis-fork/wiki/Transactions ) Reads from
outside transactions would just get a value from the latest committed
snapshot.

The unknown in this approach is how fast is it for single threaded
applications. I don't see any reasons for it too especially slow, but
it could be a bit slower than current implementation. It might be
faster too though. If anyone could work on performance measuring for
Trellis I'd really appreciate that.

Also, one more thing to mention in regards to thread safety -- even
with the GIL-like fix in place the controller has to be thread-local
because even while there's only one concurrent transaction possible
not all operations on controller are transactional: ctrl.active checks
in other threads will happen outside transactions and need to return
False.

On 2009-06-22, Sergey Schetinin <maluke at gmail.com> wrote:
> I've just brought the ForkChanges page up to date.
>  http://code.google.com/p/trellis-fork/wiki/ForkChanges
>
>  Any feedback on changes (also the "future changes" section) is welcome.
>
>
>  Also the "Transaction" page have grown into a quite long overview of
>  what's similar between Trellis rules and concurrent transactions and
>  some ideas on how Trellis could evolve. Still incomplete, but about
>  2/3 done, I think. If you see any mistakes in terminology or factual
>  errors, please report.
>
> http://code.google.com/p/trellis-fork/wiki/Transactions
>
>
>
>
>
> On 2009-06-17, Sergey Schetinin <maluke at gmail.com> wrote:
>  > On 2009-06-12, Sergey Schetinin <maluke at gmail.com> wrote:
>  >  > This is a thread for trellis-fork updates and discussion. At least it
>  >  >  should suffice at first.
>  >  >
>  >  >  http://code.google.com/p/trellis-fork/
>  >  >  http://pypi.python.org/pypi/Trellis-fork
>  >
>  >
>  > I've already finished quite a lot of work and test finally pass (had
>  >  to make a few edits, see commit log) and the changes to the code
>  >  structure are bigger than I imagined at first, but I'm quite happy
>  >  with the state of things overall.
>  >
>  >  Anyway, I'll work on it some more before reporting on the progress, so
>  >  if you want to watch the changes as they happen see
>  >  http://code.google.com/p/trellis-fork/updates/list
>  >  or the rss feed: http://code.google.com/feeds/p/trellis-fork/updates/basic
>  >
>  >  The actual reason for this email is that I want to get some feedback
>  >  on an article I started to write, explaining the use of transactions
>  >  in Trellis, it's work in progress, but it's something I think that can
>  >  help a lot with understanding Trellis.
>  >
>  >  http://code.google.com/p/trellis-fork/wiki/Transactions
>  >  Does it help?
>  >  Should it be more verbose or the maybe the opposite?
>  >  Any factual mistakes, typos, hints on structure, things to mention etc?
>  >
>  >
>  >
>  >
>  >  --
>  >  Best Regards,
>  >  Sergey Schetinin
>  >
>  >  http://s3bk.com/ -- S3 Backup
>  >  http://word-to-html.com/ -- Word to HTML Converter
>  >
>
>
>  --
>  Best Regards,
>  Sergey Schetinin
>
>  http://s3bk.com/ -- S3 Backup
>  http://word-to-html.com/ -- Word to HTML Converter
>


-- 
Best Regards,
Sergey Schetinin

http://s3bk.com/ -- S3 Backup
http://word-to-html.com/ -- Word to HTML Converter