Local scope vs relative imports inside __init__.py

Question

I've noticed that asyncio/init.py from python 3.6 uses the following construct:

from .base_events import *

...

__all__ = (base_events.__all__ + ...)

The base_events symbol is not imported anywhere in the source code, yet the module still contains a local variable for it.

I've checked this behavior with the following code, put into an __init__.py with a dummy test.py next to it:

test = "not a module"
print(test)

from .test import *
print(test)

not a module
<module 'testpy.test' from 'C:\Users\MrM\Desktop\testpy\test.py'>

Which means that the test variable got shadowed after using a star import.

I fiddled with it a bit, and it turns out that it doesn't have to be a star import, but it has to be inside an __init__.py, and it has to be relative. Otherwise the module object is not being assigned anywhere.

Without the assignment, running the above example from a file that isn't an __init__.py will raise a NameError.

Where is this behavior coming from? Has this been outlined in the spec for import system somewhere? What's the reason behind __init__.py having to be special in this way? It's not in the reference, or at least I couldn't find it.

Answer 1

This behavior is defined in The import system documentation section 5.4.2 Submodules

When a submodule is loaded using any mechanism (e.g. importlib APIs, the import or import-from statements, or built-in import()) a binding is placed in the parent module’s namespace to the submodule object. For example, if package spam has a submodule foo, after importing spam.foo, spam will have an attribute foo which is bound to the submodule.

A package namespace includes the namespace created in __init__.py plus extras added by the import system. The why is for namespace consistency.

Given Python’s familiar name binding rules this might seem surprising, but it’s actually a fundamental feature of the import system. The invariant holding is that if you have sys.modules['spam'] and sys.modules['spam.foo'] (as you would after the above import), the latter must appear as the foo attribute of the former.

Answer 2

This appears to have everything to do with the interplay of how the interpreter resolve variable assignments as the module/submodule level. We may be able to acquire additional information if we instead interrogate what the assignments are using code executed outside the module we are trying to interrogate.

In my example, I have the following:

Code listing for src/example/package/module.py:

from logging import getLogger
__all__ = ['fn1']
logger = getLogger(__name__)

def fn1():
    logger.warning('running fn1')
    return 'fn1'

Code listing for src/example/package/__init__.py:

def print_module():
    print("`module` is assigned with %r" % module)

Now execute the following in the interactive interpreter:

>>> from example.package import print_module
>>> print_module()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/example.package/src/example/package/__init__.py", line 2, in print_module
    print("`module` is assigned with %r" % module)
NameError: name 'module' is not defined

So far so good, the exception looks perfectly normal. Now let's see what happens if example.package.module gets imported:

>>> import example.package.module
>>> print_module()
`module` is assigned with <module 'example.package.module' from '/tmp/example.package/src/example/package/module.py'>

Given that relative import is a short-hand syntax for the full import, let's see what happens if we modify the __init__.py to contain the absolute import rather than relative like what was just done in the interactive interpreter and see what happens now:

import example.package.module
def print_module():
    print("`module` is assigned with %r" % module)

Launch the interactive interpreter once more, we see this:

>>> print_module()
`module` is assigned with <module 'example.package.module' from '/tmp/example.package/src/example/package/module.py'>

Note that __init__.py actually represents the module binding example.package, an intuition might be that if example.package.module is imported, the interpreter will then provide an assignment of module to example.package to aid with the resolution of example.package.module, regardless of absolute or relative imports being done. This seems to be a particular quirk of executing code at a module that may have submodules (i.e. __init__.py).

Actually, one more test. Let's see if there is just something weird to do with variable assignments. Modify src/example/package/__init__.py to:

import example.package.module

def print_module():
    print("`module` is assigned with %r" % module)

def delete_module():
    del module

The new function would test whether or not module was actually assigned to the scope at __init__.py. Executing this we learn that:

>>> from example.package import print_module, delete_module
>>> print_module()
`module` is assigned with <module 'example.package.module' from '/tmp/example.package/src/example/package/module.py'>
>>> delete_module()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/example.package/src/example/package/__init__.py", line 7, in delete_module
    del module
UnboundLocalError: local variable 'module' referenced before assignment

Indeed, it wasn't, so the interpreter is truly resolving the reference at module through the import system, rather than any variable that got assigned to the scope within __init__.py. So the prior intuition was actually wrong but it is rather the interpreter resolving the module name within example.package (even if this is done inside the scope of __init__.py) through the module system once example.package.module was imported.

I haven't looked at the specific PEPs that deals with assignment/name resolutions for modules and imports, but given that this little exercise proved that the issue is not simply reliant on relative imports, and that assignment is triggered regardless when or where the import was done, there might be something there, but this hopefully provided a greater understanding of how Python's import system deals with resolving names relating to imported modules.