CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
Medo
Medo

Reputation: 1032

multiprocessing for class method

I want to use multiprocessing for a class method. I found out from this answer that Pool in multiprocessing cannot pickle class methods directly but there is a workaround for that by defining a function outside the class, and adds an additional argument(s) to that function (Similar suggestion is also on this blog). Hence, I tried to achieve that by the following simple program which has MyClass where I want to parallel fun. However, I am not getting any results (there is no bug). It seems I am missing something but I feel I am almost there! Any fix is really appreciated.

import multiprocessing


class MyClass:
    def __init__(self):
        pass

    def fun(self, myList):
        print myList


def unwrap_fun(obj, myList):
    return obj.fun(myList)


obj = MyClass()
mlp = multiprocessing.Pool(processes=multiprocessing.cpu_count())
mlp.imap_unordered(unwrap_fun, (obj, range(1, 10)))

Upvotes: 0

Views: 3674

Answers (1)

jq170727
jq170727

Reputation: 14725

You should call close() and join() from your main process. Try this:

import multiprocessing

class MyClass:
    def fun(self, myList):
        print myList

def unwrap_fun(myList):
    obj = MyClass()
    return obj.fun(myList)

if __name__ == '__main__':
    mlp = multiprocessing.Pool(processes=multiprocessing.cpu_count())
    mlp.imap_unordered(unwrap_fun, range(1, 10))
    mlp.close()
    mlp.join()

Upvotes: 3

Related Questions