CODEWITHSUNDEEP
pythonpython-3.xpython-multiprocessingpdb
Maxander
Maxander

Reputation: 405

Multiprocessing code fails when run with pdb?

Related to Python Multiprocessing error: AttributeError: module '__main__' has no attribute '__spec__' , but arising from different circumstances.

I'm encountering an issue in Python 3.7.4 when I try to run multiprocessing code with pdb. The issue replicates with the basic multiprocessing example from https://docs.python.org/3.6/library/multiprocessing.html :

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

This runs fine (outputs [1, 4, 9]) when run directly from Python via python.exe testcase.py. However, it does not work under pdb; python.exe -m pdb testcase.py fails with an error:

Traceback (most recent call last):
  File "c:\python37\lib\pdb.py", line 1697, in main
    pdb._runscript(mainpyfile)
  File "c:\python37\lib\pdb.py", line 1566, in _runscript
    self.run(statement)
  File "c:\python37\lib\bdb.py", line 585, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "c:\users\max\desktop\projects\errortest.py", line 1, in <module>
    from multiprocessing import Pool
  File "c:\python37\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "c:\python37\lib\multiprocessing\pool.py", line 176, in __init__
    self._repopulate_pool()
  File "c:\python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
    w.start()
  File "c:\python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "c:\python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "c:\python37\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "c:\python37\lib\multiprocessing\spawn.py", line 172, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\python37\lib\multiprocessing\spawn.py(172)get_preparation_data()
-> main_mod_name = getattr(main_module.__spec__, "name", None)

I hesitate to think that I've found a bug in a pair of modules that have been important parts of Python for over a decade. Is something incorrect here?

Upvotes: 6

Views: 1807

Answers (1)

tersrth
tersrth

Reputation: 889

This is a limitation of multiprocessing in windows. This question contains a good explanation for why this is so. A quick google search shows that the puDB may be able to help with debugging multi-processing code, but I have not used it before.

The following is from the python docs:

Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.pool.Pool examples will not work in the interactive interpreter.

Upvotes: 3

Related Questions