CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessingspyder
user8474060
user8474060

Reputation: 879

Python Multiprocessing error: AttributeError: module '__main__' has no attribute '__spec__'

I'm using Python 3.6 and am trying to follow along with the very first example at the website below (full code also below) and am getting the below error: https://docs.python.org/3.6/library/multiprocessing.html

Error message: AttributeError: module '__main__' has no attribute '__spec__'

Full example code:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

I tried Googling it and searching Stack Overflow but I've only found one other case of this error and it did not have an answer.

Upvotes: 46

Views: 39476

Answers (6)

kluonk
kluonk

Reputation: 115

I think the best fix or workaround has already been posted, but the real origin was not clarified. And I stumbled upon this issue not in Spyder. So first of all, I would agree with @user8474060's answer except for @Kevin modification:

if __name__ == '__main__':
    __spec__ = None
    with Pool(5) as p:
       print (p.map(f, [1, 2, 3]))

But the origin of the problem is not bound to spyder, nor to multiprocessing but rather an interplay of the two, or more specific of IPython (not spyder). There are a few things to consider:

  • multiprocessing does not work in an interactive interpreter, as discussed in the comments of this stack post and explained in the docs (here, only linked to ProcessPoolExecutor, which is jsut a wrapper)
  • So you have to use it within a python script, e.g. example.py. As already mentioned in @user8474060's answer, it works nicely when running from a system terminal via e.g. python example.py
  • When running a script in Spyder it will run it by default within the current IPython console via runfile, see this stack answer. This does not execute another instance of python as a pure python example.py in an external terminal would do.
  • Note that this Run command can be configured to run the file in a dedicated console or an external terminal, as mentioned already by Simin Zuo. However, I don't understand why I didn't work for Simin Zuo with the external terminal.
  • Regarding switching between External terminal and Execute in current console, it should always work when running the script for the first time in IPython via runfile, so if it is the first input to the kernel, showing something like In [1]: runfile(/path/to/example.py). It will break upon the second run.
  • Now, I would like to mention that it does not affect Spyder, but all IPython systems. I stumbled upon this issue using Jupyter notebooks. It does not matter whether it's used in jupyer lab, jupyter notebook, or any other IDE supporting jupyter notebook.
  • In order to run multiprocessing scripts from within jupyter I ran %run example.py. Similar like spyder's runfile the magic command runs the file within the current ipython kernel.
  • For jupyter itself it might just be a simple solution to run it as !python example.py, running a fresh python instance from the system shell

Upvotes: 1

DrStrangepork
DrStrangepork

Reputation: 3144

I stumbled on this question in researching this error. I found that running

with Pool() as mp_pool:

caused the error. Changing this to:

if __name__ == '__main__':
    __spec__ = None
    with Pool() as mp_pool:

resolved it. Python 3.11.4. Fix had nothing to do with Spyder or Anaconda.

Upvotes: 0

kiwi_kimchi
kiwi_kimchi

Reputation: 423

Same problem with Spyder (Anaconda3, python 3.7).

I used

from genetic_selection import GeneticSelectionCV

def main(): .... and as I was running the code, an error like this occured:

main_mod_name = getattr(main_module.__spec__, "name", None)

AttributeError: module '__main__' has no attribute '__spec__'

what I did is deleted "__spec__" in main_mod_name = getattr(main_module.__spec__, "name", None)

so I only have this: main_mod_name = getattr(main_module, "name", None)

the code then worked perfectly fine.

Upvotes: 0

JoeyC
JoeyC

Reputation: 824

pdb users

The question didn't specifically mention Spyder nor Conda (though it is tagged as such). Hence, I will note that I found this can also happen when using pdb.

E.g.

python -m pdb myprogram.py

Passing __spec__ = None would be a useful workaround if you wanted to persist with pdb.

Upvotes: 37

Simin Zuo
Simin Zuo

Reputation: 21

the same probelm in Spyder (Anaconda3, python 3.6) when I try the external terminal.

Error message: AttributeError: module '__main__' has no attribute '__spec__'

I changed the Run console to 'Excute in current console', and applied it. then if that doesnot work, try other conselor and then change back to 'Excute in current console'. Finally, it works. no '__spec__ = None' is needed.

Upvotes: 1

user8474060
user8474060

Reputation: 879

The problem is not with the code / Python 3.6, it is with Spyder.

After some investigation I found that the code runs fine when executed in an external system terminal but not when run in Spyder's IPython console.

I was able to dump the contents of spec and assign them to a variable that was included inside main to allow this code to function within the IPython console.

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    __spec__ = "ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>)"
    with Pool(5) as p:
       print (p.map(f, [1, 2, 3]))

Upvotes: 41

Related Questions