CODEWITHSUNDEEP
pythonpicklepython-multiprocessing
vks
vks

Reputation: 67968

MultiProcessing with self.methods

Using multiprocessing on methods we get error stating self.methods cannot be pickled.

To overcome this I used:

def _pickle_method(m):
    if m.im_self is None:
        return getattr, (m.im_class, m.im_func.func_name)
    else:
        return getattr, (m.im_self, m.im_func.func_name)


copy_reg.pickle(types.MethodType, _pickle_method)

Now I searched about this, but few questions are still unclear:

  1. Why can't self.methods be pickled?
  2. How does copy_reg.pickle() work? How does it enable pickling of self.methods?
  3. Are there any negatives of using this approach or are there any other, better, methods?

More Info:

I had a function in a class which used to do a request.get and request.post. To improve times , I used multiprocessing over it. This is the exact problem I faced.

Can't pickle <type 'instancemethod'> when using multiprocessing Pool.map()

Upvotes: 3

Views: 5457

Answers (1)

martineau
martineau

Reputation: 123463

Your intended usage is still a little vague—so I will instead show a way to solve the linked question in the new "More info" section of your question that doesn't require the pickling the byte-code of a class method. I think how it does it is fairly obvious...

someclass.py

import multiprocessing

def call_method(x):
    return SomeClass().f(x)

class SomeClass(object):
    def __init__(self):
        pass

    def f(self, x):
        return x*x

    def go(self):
        pool = multiprocessing.Pool(processes=4)
        print(pool.map(call_method,  range(10)))
        pool.close()

test.py

import someclass

if __name__== '__main__' :
    sc = someclass.SomeClass()
    sc.go()

Output:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Upvotes: 3

Related Questions