CODEWITHSUNDEEP
pythonfor-loopmultiprocessing
McCuys
McCuys

Reputation: 37

Parallelize for loop inside a method

I would like to parallelise the for loop in fun2() of the following code but I cannot figure out how. I would like to keep the function inside of class2 and that seems to be why mapping to multiprocessing.Pool does not work. But I was also unable to make it work with multiprocessing.pool.ThreadPool.

class class1():
    def __init__(self, x):
        self.var1 = x

    def fun1(self):
        self.result = self.var1**2

class class2():
    def __init__(self, y):
        self.var2 = y

    def fun2(self):
        for v in self.var2:
            obj = class1(v)
            obj.fun1()
            setattr(self, 'obj_{}'.format(v), obj)


obj2 = class2(range(5))
obj2.fun2()

This code works but is very slow in my case as fun1 is very expensive in my real programm. Any help is very much appreciated. Thank you!

Upvotes: 0

Views: 60

Answers (1)

Roy2012
Roy2012

Reputation: 12493

Here's a way to do that. The change is in class2

class class1():
    def __init__(self, x):
        self.var1 = x

    def fun1(self):
        time.sleep(1)              #simulate a long-running job
        self.result = self.var1**2
        print (self.result)

class class2():
    def __init__(self, y):
        self.var2 = y

    def _sub_process(self, v):
        obj = class1(v)
        obj.fun1()
        setattr(self, 'obj_{}'.format(v), obj) 
        print(f"in _subprocess, v is {v}")

    def fun2(self):
        with multiprocessing.Pool() as p:
            print(p.map(self._sub_process, self.var2))        

if __name__ == "__main__":
    obj2 = class2(range(5))
    obj2.fun2()

Upvotes: 1

Related Questions