CODEWITHSUNDEEP
python-3.xpython-multiprocessingpython-3.8
Btc Sources
Btc Sources

Reputation: 2061

multiprocessing.queue module is missing until a Queue is instanced

I was working with Queues and everything seemed to go fine. However, I found a situation that makes an unexpected AttributeError exception to arise.

Let's create the simplest code to create a Queue. Don't worry about it's usage, it all happens in the same side, we don't have to worry about the other side that will communicate with it.

import multiprocessing

q = multiprocessing.Queue()
print(type(q))

This code will produce as output:

<class 'multiprocessing.queues.Queue'>

But what happens if I try to identify if the object is an instance of this class?

isinstance(q, multiprocessing.queues.Queue)

Obviously it returns True.

Finally, my question arose due to that, if I don't create a Queue instance before that checking, the module multiprocessing.queues where Queue is defined, is not loaded! (Warning, maybe loaded is not the appropriate word here since we are talking about modules. I'm not an expert on it) Let's check it:

import multiprocessing

a = dict()
isinstance(a, multiprocessing.queues.Queue)

Where surprisingly we get the failure:

 1 isinstance(a, multiprocessing.queues.Queue)

AttributeError: module 'multiprocessing' has no attribute 'queues'

Why is this happening? Is this the intended behaviour or is this kind of a bug?

For me, this represents a problem because in case I want to check if an object is a Queue and it's not, if no other queue has been instanced my check could end in an undesired AttributeError.

Finally, I would like to add a quick picture with some checks that I did reproducing this, plus adding some checks for the queues module inside the multiprocessing.__all__.

These checks have been performed in Python 3.9.1. It is worth noting that, when multiprocessing.queues was already recognized and therefore the checks didn't throw the AttributeError, queues was present in multiprocessing.__dict__. In the other cases, it is not.

This may seem natural to the expert eye, since __dict__ is (AFAIK) a list of attributes, but I wanted to point it out since as I said I'm not an expert and I don't know the actual implications of it.

enter image description here

PS. I've tagged the question as Python 3.8 since it has been tested in both 3.8 and 3.9 with same results, and in case there are differences I would like the answer for 3.8. Those differences are likely to appear due to multiprocessing library changes as the docs reflects in several footnotes at the end of the functions.

PS.2. I've been referring to multiprocessing.queues as a module due to the output of multiprocessing.__dict__. Again I'm not an expert on the matter:

'queues': <module 'multiprocessing.queues' from 'C:\\Program Files\\Python39\\lib\\multiprocessing\\queues.py'>

Upvotes: 1

Views: 958

Answers (1)

MisterMiyagi
MisterMiyagi

Reputation: 52149

This is intended behaviour. If you want to access a submodule, you should import it:

import multiprocessing.queues

a = dict()
isinstance(a, multiprocessing.queues.Queue)

Importing submodules adds them as attributes to their parent module – note how multiprocessing.queues.Queue is a nested attribute lookup of Queue on queue after queue on multiprocessing. An explicit import ensures the submodule will be loaded before accessing it.

The import system: 5.4.2. Submodules

When a submodule is loaded using any mechanism (e.g. importlib APIs, the import or import-from statements, or built-in __import__()) a binding is placed in the parent module’s namespace to the submodule object. For example, if package spam has a submodule foo, after importing spam.foo, spam will have an attribute foo which is bound to the submodule.

This means one can get away skipping an explicit export when another action imports the submodule as a side effect. As with all side effects, one has to actually perform the triggering action in order to rely on the side effect.

Since imports are idempotent (repeating them has no ill effect), one should import any module one expects to access instead of relying on side effects of other code.

Upvotes: 3

Related Questions