CODEWITHSUNDEEP
pythonmultiprocessingpickle
Amir
Amir

Reputation: 6186

Multiprocessing: Pool and pickle Error -- Pickling Error: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

I have two files:

x.py

class BF(object)
   def __init__():
   .
   .
   def add(self,z):
   .
   .

y.py

from y import BF
def FUNC((a,b,bf))
   .
   .
   bf.add(x)
   .
   .
   return bf

.
.
if __name__ == '__main__':
    pool = multiprocessing.Pool(3)
    for i in range(len(sl)):
         bf_set.append(BF())
    results = pool.map(FUNC,zip(sl, itertools.repeat(aa), bf_set))

I also tried to define BF inside FUNC, but sill I got: 

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

I've read some posts for related issues, but they have their pool.map() inside of the class, so the solutions cannot be applied to this problem (i guess).

Any idea?

Upvotes: 1

Views: 4477

Answers (1)

Mike McKerns
Mike McKerns

Reputation: 35217

I'm going to basically use what you have above, but turn it into working code. There is no problem serializing, if you use dill. I'm using a fork of multiprocessing called pathos.multiprocessing, which uses dill instead of pickle.

>>> def FUNC((a,b,bf)):
...   z = a+b
...   bf.add(z)
...   return bf
... 
>>> class BF(object):
...   def add(self, z):
...     self.z += z
...   def __init__(self):
...     self.z = 0
... 
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> pool = Pool()
>>> 
>>> f = BF()
>>> f.add(1)
>>> f.z
1
>>> 
>>> FUNC((0,1,f))
<__main__.BF object at 0x10d387f50>
>>> 
>>> FUNC((0,1,f)).z
2
>>> 
>>> sl = [BF() for i in range(10)]
>>> results = pool.map(FUNC, zip(range(len(sl)), range(len(sl)), sl))
>>> [bf.z for bf in results]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

This works, because pathos uses dill, which can serialize almost anything in python.

>>> import dill as pickle
>>> pickle.loads(pickle.dumps(bf.add))
<bound method BF.add of <__main__.BF object at 0x10d383950>>
>>> pickle.loads(pickle.dumps(BF.add))
<unbound method BF.add>

Get pathos and dill at: https://github.com/uqfoundation

Upvotes: 1

Related Questions