Are python circular imports an implementation detail?

Question

Is the behaviour of circular imports well-specified in Python, or is it just implementation-specific?

For example in cpython, importing a submodule causes the submodule to be assigned as an attribute of the parent module, but not until after the submodule finishes executing. Is this stated anywhere in the docs or PEPs?

More often then not, python circular imports just work (assisted by conventions like not executing any functions during import, and laying out the source code to define functions in the reverse of expected call order), but a consequence of the above behaviour is that you cannot refer by parent to a partially imported module. For example, if importing package.submodule causes an import of othermodule, then othermodule cannot do import package.submodule as submod but can do from package import submodule as submod. Is this explained/specified by the documentation? If not, is it likely to change?

In this case, the discrepancy is because import package.submodule as submod is implemented as a call to __import__ (that returns package) followed by an attribute lookup (to retrieve submodule so that it can be assigned to submod). In contrast, from package import submodule as submod causes __import__ to directly return submodule (which seems to work even if submodule is still only half-complete). You could figure some of this out if you delve through the cpython bytecode (documented in the dis section of the standard library) and study the __import__ semantics. However, since resolving circular imports is a common issue in python, it would be useful to find an official high-level summary of what to expect?

Example:

mkdir package
touch package/__init__.py
cat > package/submodule.py << EOF
def f():
  print('OK')
import othermodule
othermodule.g()
def notyetdefined():
  pass
EOF
cat > othermodule.py << EOF
def g():
  try:
    import package.submodule as submod # this fails
  except AttributeError:
    print('FAIL')
  from package import submodule as submod # this works ok
  submod.f()
  #submod.notyetdefined() # this cannot be invoked
EOF
python -c 'import package.submodule'

Output in python 3.6.7:

FAIL
OK

Answer 1

From a little research, it sounds like the answer is that there is a bit of both specification and undocumented behavior involved, relating to how modules are initialized and how different forms of the import statement resolve (sub)modules. Overall, it looks like the behavior of circular imports should be fairly well-defined by the system, but the behavior you saw was "an implementation quirk."¹

While I didn't investigate the rules of Python's import system in great detail, I was able to get to the bottom of the particular issue you observed.

To get there, I first noticed that, the behavior of your code changed in Python 3.7: it now only prints OK. This point from the changelog for Python 3.7 says why:

Circular imports involving absolute imports with binding a submodule to a name are now supported. (Contributed by Serhiy Storchaka in bpo-30024.)

The discussion on the Python issue that led to the change contains some discussion of what was going on before, as well as a handful of links to earlier discussions of why the pre-3.7 behavior worked why it did. I found a few of the comments and links to be particular useful here:

The very first comment gives an explanation for the the behavior you observed (this Stack Overflow answer discusses how the same error occurs in a similar case):

The background here is the change in http://bugs.python.org/issue17636 that allows IMPORT_FROM to fall back to sys.modules when written as "from a.b import c as m", while the plain LOAD_ATTR generated for "import a.b.c as m" fails.

Note that bpo-17636 motivated the support of "[c]ircular imports involving relative imports" in Python 3.5.

More generally, for your answer, this comment (from Guido himself) states that the behavior of circular imports is defined:

The semantics of imports in the case of cycles are somewhat complex but clearly defined and there are only a few rules to consider, and from these rules it is possible to reason out whether any particular case is valid or not.

Based on the fact that neither "circle," "cycle," nor any of the obvious variants thereof appear in the documentation for the import system, I assume that the rules regarding cyclic imports, while consistent, are emergent properties of the system rather than explicit behaviors.

(Note that the documentation for the import system also doesn't mention the changes in 3.7, and doesn't mention anything about import ... as ... or from ... import ... at all. While the documentation for the import statement does discuss the different forms of the statement, it doesn't discuss cycles either (and most of it hasn't been updated in at least 3 years).)

---

There's also one small change coming in Python 3.8, though it does not appear to be in the changelog.

If you uncomment submod.notyetdefined() in othermodule, Python 3.6 and 3.7 both raise the following error:

AttributeError: module 'package.submodule' has no attribute 'notyetdefined'

In Python 3.8.0b4, this more useful message is produced instead:

AttributeError: partially initialized module 'package.submodule' has no attribute 'notyetdefined' (most likely due to a circular import)

This message appears to be the result of a simple check whether the current module is being initialized when a missing attribute was accessed; it's doesn't actually do anything relating to circular exports, except know that they are the most likely reason that an undefined attribute of a module might be accessed while it is being initialized.

---

¹ Ironically, that quote is from this email, which is given in the discussion for bpo-30024 as the rationale for the changes arising from bpo-17636—they fixed one case, but left another as it was, and that's what caused your error.