[PEAK] imports (Re: Package organization)

[Phillip J. Eby]

At 07:30 PM 12/4/03 +0200, alexander smishlajev wrote:
>Phillip J. Eby wrote, at 04.12.2003 16:13:
>>Second, if it doesn't export the primitives (especiall NOT_GIVEN
>>and NOT_FOUND), where will you get them from?
>
>and that's because i misread what the primitives are.  somehow protocols 
>and adapt captured my attention.  protocols is an independent module, and 
>my personal preference is to import it explicitely, not to get as a bonus 
>from peak core api.  of course, this makes sense only if the export 
>semantics change from "get all of PEAK" to "get things required to work 
>with PEAK", i.e. if frameworks and other components are not exported by 
>default.

The plan at this point is for 'peak.core' to be a subset of today's 
'peak.api', such that only these names:

Items
NOT_GIVEN
NOT_FOUND
PropertyName
adapt
protocols
binding
config
naming

would be exported.  'peak.api', OTOH, would continue to expand as the 
available frameworks grow, but would not include "consolidation packages" 
like peak.tools, 'peak.metamodels', or 'peak.util'.



>>>first, there is inconsistence between LazyModule and _loadAndRunHooks: 
>>>if any of the hooks fail, _loadAndRunHooks still believes that the 
>>>module is imported (by disabling all postLoadHooks for that module), but 
>>>LazyModule thinks that the module is *not* imported, and tries to 
>>>_loadAndRunHooks again upon the next attribute access.  this leads to 
>>>AlreadyRead exception.
>>True.  And then the module concludes it's *still* not imported.
>>Unfortunately I don't think there's much I can do about this, except 
>>maybe allow the hooks to be run more than once, which I don't really like.
>
>maybe just remove hooks that ran successfully?  and then isolate hooks 
>effects by taking a copy of the module __dict__ and restore it if the hook 
>raised an exception?
>
>i am not sure if such approach is implementable; these are just crazy ideas.

I'm tempted to have such errors cause a fatal interpreter error, because 
proceeding sanely at that point is pretty much impossible, and rollback is 
also impossible.  There are just too many potential complications from 
side-effects of module import, such as modules initializing registries or 
calling other modules to register things.

Indeed, I'm beginning to realize that what it does now may actually be a 
*feature*, not a bug.  Making it impossible to access the module makes it 
pretty plain that the environment is hosed.  The only way to fix it for 
certain is to rollback the import of every module that imported the broken 
module, along with every module that imported any of those modules.  That 
is, almost nothing short of an interpreter restart.  (And if any of those 
modules modified other modules directly, those modified modules would have 
to be rolled back, too!)


>>>     module = imp.new_module(fqname)
>>>     exec code in module.__dict__
>>>
>>>this launches _loadAndRunHooks upon access to module.__dict__ (or any 
>>>special import attribute, like '__importer__' or '__ispkg__'), when the 
>>>module code has not been executed, and always leads to the problem shown 
>>>in above example.
>>I'm a little puzzled as to how they get to that point, unless 
>>imp.new_module is returning the module from sys.modules.
>
>it is not.  the module created by imp.new_module does not appear in 
>sys.modules.

But if that's the case, then how is it that this code is touching a lazily 
imported module?  If it's a new module, it can't be the lazy one, so how 
can it be touching the module dictionary?  That's the part I don't understand.


>>I suppose I could make the __getattribute__ avoid loading on access to 
>>attributes that begin and end with double underscores.
>
>this may behave somewhat better, but still there are things like 
>config.interfaces accessing own IConfigKey via indirect recursion in the 
>very first line of code, when IConfigKey is not created yet.

This is the part I don't understand.  I would guess from what you describe, 
that when using imputil, reload() is attempting to access the existing 
module.  But, from what I can tell in the imputil source, imputil doesn't 
*hook* reload().

Is imputil in fact the package giving you this trouble?  If so, how?  And 
what do you use imputil for, anyway?



>>But, it's possible that for some attributes, this would be a bad 
>>thing.  For example, a module's __conform__ method would be accessed by 
>>adapt().  Without a comprehensive list of which attributes to ignore, it 
>>would be unlikely to be correct in all cases.  I could add such a list of 
>>attributes known to be read by importers, I suppose.
>
>i am pretty sure about three names: "__dict__", "__ispkg__" and 
>"__importer___".  i am not saying that i want this ASAP; as i said before, 
>we already found a solution acceptable for us.

I understand that.  I just don't understand yet how this is breaking.  I 
understand what you've described as the symptom, I just don't understand 
how you can get to a place where that can occur, given what I know so far.


>i use Analysis from McMillian installer to find most of used modules. some 
>modules are not found this way, and i list those names in a modulefinder 
>hook as 'hiddenimports'.  three PEAK modules still cannot be found by 
>Analysis: peak.binding.api, peak.running.api, peak.interface; i add them 
>to the result TOC manually.

peak.interface probably doesn't show up because it isn't used anymore, or 
at least shouldn't be.  I've been meaning to take it out, so this is a good 
excuse for me to go ahead and do it.  :)

As for the others, I imagine I could add some code to peak.api that does:

import peak.binding.api as binding
import peak.naming.api as naming
# ... etc.

After the lazy imports.  This would ensure that the imports get found by 
any analysis tool that looks for import bytecodes.



>if you are interested, i may pass all sources involved in the process: 
>installer .spec file, module hook, executable stub and the script making 
>an executable from this stub.

I expect this would be of interest to many people.  Perhaps you could place 
them on the wiki, if you don't have a place to publish them from 
yourself?  I don't want to add them to the PEAK distribution until/unless I 
can also support them, but publishing them on the wiki or linking them from 
the Wiki would be a good idea for making the information available.  Thanks.