CODEWITHSUNDEEP
pythonmultiprocessingpickle
ruben baetens
ruben baetens

Reputation: 2926

Multiprocessing results in PicklingError

Executing the example 

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

from the python documentation for the package multiprocessing results in a PicklingError: Can't pickle <function f at 0x05A57830>: it's not found as __main__.f.

How can i solve this ?

the complete error message is

Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
  File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\startup.py", line 128, in runfile
    execfile(filename, glbs)
  File "C:\Users\Ruben\Desktop\untitled0.py", line 15, in <module>
    p.start()
  File "C:\Python27\lib\site-packages\multiprocessing-2.6.2.1-py2.7-win32.egg\multiprocessing\process.py", line 109, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\site-packages\multiprocessing-2.6.2.1-py2.7-win32.egg\multiprocessing\forking.py", line 244, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\site-packages\multiprocessing-2.6.2.1-py2.7-win32.egg\multiprocessing\forking.py", line 167, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function f at 0x05A57830>: it's not found as __main__.f

Upvotes: 2

Views: 1648

Answers (1)

unutbu
unutbu

Reputation: 879471

This is a known bug in IPython. You are running the code from the ipython console:

File "<ipython console>", line 1, in <module>

where __main__ references a FakeModule and pickle can not find __main__.f :

PicklingError: Can't pickle <function f at 0x05A57830>: it's not found as __main__.f

(multiprocessing uses pickle to pass functions and arguments to the worker processes.)

So the solution is to run your code as a script, not from ipython.

Upvotes: 4

Related Questions