CODEWITHSUNDEEP
pythonasynchronousmultiprocessing
David Parks
David Parks

Reputation: 32061

multiprocessing.apply_async(myfunc, ...) within a class doesn't seem to call myfunc

import multiprocessing


class multiprocessing_issue:
    def __init__(self):
        self.test_mp()

    def print_test(self):
        print "TEST TEST TEST"

    def test_mp(self):
        p = multiprocessing.Pool(processes=4)
        p.apply_async(self.print_test, args=())
        print "finished"


if __name__ == '__main__':
    multiprocessing_issue()

I've set up a simple test above, create a class, call apply_async with a function that should print "TEST TEST TEST". When I run this I see "finished" printed, but it never prints "TEST TEST TEST" as expected.

Can anyone see the error in this simple test case? I've set it up to reproduce the way I'm using it in my code.

Python 2.7 on Ubuntu

Upvotes: 1

Views: 3333

Answers (2)

noxdafox
noxdafox

Reputation: 15040

Modify test_mp as follows:

def test_mp(self):
    p = multiprocessing.Pool(processes=4)
    r = p.apply_async(self.print_test, args=())
    print r.get()

and the answer will be more clear.

Traceback (most recent call last):
  File "test.py", line 18, in <module>
    multiprocessing_issue()
  File "test.py", line 6, in __init__
    self.test_mp()
  File "test.py", line 14, in test_mp
    print r.get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Instance methods cannot be serialized that easily. What the Pickle protocol does when serialising a function is simply turning it into a string.

In [1]: dumps(function)
Out[1]: 'c__main__\nfunction\np0\n.'

For a child process would be quite hard to find the right object your instance method is referring to due to separate process address spaces.

Modules such as dill are doing a better job than Pickle. Yet I would discourage you from mixing concurrency and OOP as the logic gets confusing pretty easily.

Upvotes: 4

David Parks
David Parks

Reputation: 32061

Ah, it's a problem moving the class reference between processes, if I define the method at the module level instead of the class level everything works.

import multiprocessing


class multiprocessing_issue:
    def __init__(self):
        self.test_mp()

    def test_mp(self):
        p = multiprocessing.Pool(4)
        r = p.apply_async(mptest, args=())
        r.get()
        print "finished"


def mptest():
    print "TEST TEST TEST"


if __name__ == '__main__':
    multiprocessing_issue()

Upvotes: 2

Related Questions