CODEWITHSUNDEEP
pythonmultiprocessing
Janus Gowda
Janus Gowda

Reputation: 305

Parallel execution of class methods

I need to execute in parallel a method of many instances of the same class. For doing this I'm trying to use the Process.start() and the Process.join() commands from the multiprocessing module.

For example for a class:

class test:
     def __init__(self):
     ...
     ...
     def method(self):
     ...
     ...

where method modifies some of the class variables. If I make two instances of the class:

t1=test()
t2=test()

And execute:

from multiprocessing import Process
pr1=Process(target=t1.method, args=(,))
pr2=Process(target=t2.method, args=(,))
pr1.start()
pr2.start()
pr1.join()
pr2.join()

the variables of the instances of the class are not updated (the whole code is too long to be pasted here but this is the idea).

Is there any way to achieve this? Thank you

Upvotes: 9

Views: 15221

Answers (2)

Aniket Hosamani
Aniket Hosamani

Reputation: 31

Using this logic you can call 2 methods of two different instances parallelly

+ Or you can call two different methods of SAME INSTANCES parallelly

...This might be wrong or not a good practice... please let me know in the comments :)

def parallel_call_method(inst):
    inst.method()


class Test(object):

    def __init__(self):
        self.some_list = []  # Normal list

    def method(self):
        self.some_list.append(123)  # This change won't be lost


if __name__ == "__main__":

    t1 = Test()
    t2 = Test()

    t1 = threading.Thread(target=parallel_call_method, args=(t1,))
    t2 = threading.Thread(target=parallel_call_method, args=(t2,))
    t1.start()
    t2.start()

    print(t1.some_list)
    print(t2.some_list)

Upvotes: 0

dano
dano

Reputation: 94881

When you call obj.method in a child process, the child process is getting its own separate copy of each instance variable in obj. So, the changes you make to them in the child will not be reflected in the parent. You'll need to explicitly pass the changed values back to the parent via a multiprocessing.Queue in order to make the changes take effect the parent:

from multiprocessing import Process, Queue
q1 = Queue()
q2 = Queue()
pr1 = Process(target=t1.method, args=(q1,))
pr2 = Process(target=t2.method, args=(q2,))
pr1.start()
pr2.start()
out1 = q1.get()
out2 = q2.get()
t1.blah = out1
t2.blah = out2
pr1.join()
pr2.join()

Other options would be to make the instance variables you need to change multiprocessing.Value instances, or multiprocessing.Manager Proxy instances. That way, the changes you make in the children would be reflected in the parent automatically. But that comes at the cost of adding overhead to using the variables in the parent.

Here's an example using multiprocessing.Manager. This doesn't work:

import multiprocessing

class Test(object) :

    def __init__(self):
       self.some_list = []  # Normal list

    def method(self):
        self.some_list.append(123)  # This change gets lost


if __name__ == "__main__":
    t1 = Test()
    t2 = Test()
    pr1 = multiprocessing.Process(target=t1.method)
    pr2 = multiprocessing.Process(target=t2.method)
    pr1.start()
    pr2.start()
    pr1.join()
    pr2.join()
    print(t1.some_list)
    print(t2.some_list)

Output:

[]
[]

This works:

import multiprocessing

class Test(object) :

    def __init__(self):
       self.manager = multiprocessing.Manager()
       self.some_list = self.manager.list()  # Shared Proxy to a list

    def method(self):
        self.some_list.append(123) # This change won't be lost


if __name__ == "__main__":
    t1 = Test()
    t2 = Test()
    pr1 = multiprocessing.Process(target=t1.method)
    pr2 = multiprocessing.Process(target=t2.method)
    pr1.start()
    pr2.start()
    pr1.join()
    pr2.join()
    print(t1.some_list)
    print(t2.some_list)

Output:

[123]
[123]

Just keep in mind that a multiprocessing.Manager starts a child process to manage all the shared instances you create, and that every time you access one of the Proxy instances, you're actually making an IPC call to the Manager process.

Upvotes: 16

Related Questions