CODEWITHSUNDEEP
pythonpython-3.xanacondapython-multiprocessing
hobboy
hobboy

Reputation: 50

Python 2 to Python 3: can't get multiprocessing to work

I'm moving my code from python 2.7 to python 3.5 and cannot get my multiprocessing code to work, something like the following code. "somemodule.py" in the same directory as the main script.

import multiprocessing as mp

# somemodule is in the same folder as this script
import somemodule

def foo(bar):
    print(bar)
    return

if __name__ == '__main__':
    bar = ("alice","bob")
    pool = mp.Pool(processes=2)
    pool.map(foo, bar)
    pool.close()

The traceback is

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
    prepare(preparation_data)
  File "C:\Anaconda3\lib\multiprocessing\spawn.py", line 226, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Anaconda3\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Anaconda3\lib\runpy.py", line 240, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Anaconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\hobboy\Documents\project\pool_test3.py", line 2, in <module>
    import somemodule
ImportError: No module named 'somemodule'

The code works in the Python Console of Spyder, but not IPython (working with Anaconda3 2.5.0). The worker processes are unable to import somemodule but the main process can. Is it something to do with import somemodule being inadequate, or something else? I've read into the Python 3 import syntax a bit, but I'm still not sure.

I've made a post on the Anaconda community forums but that's more in regards to if it's an issue with Anaconda rather than the code itself.

Additional information:

For main process (non worker) for Python 2.7/3.5, sys.path[0]='' regardless of console/python version

What's interesting is that sys.path[0] varies for the worker processes:

Spyder 2.3.8 (Python 2.7)

Spyder 2.3.8 (Python 3.5)

However the main (non worker) process for both Python 2 and 3 being sys.path=''. Why is sys.path[0] being set differently (and inconsistently) in Python 3

Upvotes: 0

Views: 1902

Answers (1)

Giacomo Lacava
Giacomo Lacava

Reputation: 1833

Look at what you get when you add the following code on the very first line of your script:

import sys; print(sys.path)

The output should include the directory where your "somemodule" is. If it's not listed, you can explicitly manipulate your environment by appending the directory to the sys.path list (careful escaping backslashes) before importing the module.

Another alternative is making sure the python interpreter is run directly from that folder, so that the "working directory" includes somemodule.py.

Upvotes: 3

Related Questions