CODEWITHSUNDEEP
pythonmultithreadingpython-2.7picklepathos
RHam
RHam

Reputation: 21

pathos multiprocessing cannot pickle

I am having a similar issue to this person. I am unable to run a simple multiprocessing routine in the pathos module and receive a pickling error. Below is the code and error.

from pathos.multiprocessing import ProcessingPool
import dill
class ProcClass(object):
    def __init__(self):
        pass
    def f(self,x):
        return x*x
pc = ProcClass()
pl = ProcessingPool(3)
print pl.map(pc.f, range(10))

The returned error:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/multiprocessing/pool.py", line 320, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I have installed: pp, dill, pox and multiprocessing then installed pathos. The install works but always gives me this error:

WARNING: One of the following dependencies is unresolved: pp(ft) >=1.6.4.5 dill >=0.2.4 pox >=0.2.2 (multi)processing

Based on a response from the author of pathos to a similar question, it looks like there is a problem with the install. I have removed and reinstalled several times, each time verifying the proper dependancies are installed. I'm running on MacOS and using python 2.7. Any help will be greatly appreciated!

Upvotes: 2

Views: 4642

Answers (2)

string_is_hard
string_is_hard

Reputation: 29

I was having similar error. I changed my import part from:

from pathos.multiprocessing import ProcessingPool

to:

import pathos, multiprocess
from pathos.multiprocessing import ProcessingPool
import dill

and the code works. Not sure what's happening though. Maybe like Mike mentioned in another answer that pathos is finding the system bundled multiprocessing package instead of multiprocess. And maybe explicitly importing it somehow fixed the issue...

Upvotes: 0

Mike McKerns
Mike McKerns

Reputation: 35247

It appears that you are missing a critical dependency. With pathos, you should use multiprocess, which is a fork of multiprocessing which uses the dill serializer.

>>> import multiprocessing
>>> multiprocessing.__version__
'0.70a1'
>>> import multiprocess   
>>> multiprocess.__version__
'0.70.4.dev0'
>>> 
>>> multiprocess.Pool().map(lambda x:x*x, range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>>

Upvotes: 0

Related Questions