CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
Arco Bast
Arco Bast

Reputation: 3892

executing n iterations of for loop in parallel

Let's assume we have a nested for loop in python, where the innermost loop calls some function fun:

for x1 in ['a','b','c','d','e']:
    for x2 in [1,2,3]:
        fun(x1,x2)

How could this be rewritten, such that n calls of fun are executed in parallel? I.e. for n==2, the order of execution would be:

How could this be achieved in the most pythonic way?

Upvotes: 0

Views: 97

Answers (2)

Tomerikoo
Tomerikoo

Reputation: 19432

You can use the multiprocessing module:

from multiprocessing import Pool
from itertools import product

if __name__ == "__main__":
    l1 = ['a', 'b', 'c', 'd', 'e']
    l2 = [1, 2, 3]
    n = 2
    with Pool(n) as pool:
        pool.starmap(fun, product(l1, l2))

Using product we create all the pairs in a row:

>>> list(product(['a', 'b'], [1, 2]))
[('a', 1), ('a', 2), ('b', 1), ('b', 2)]

Then, we can create a Pool with n processes, and pass each pair to fun using the starmap function.

By creating the Pool and passing the first parameter n we have n processes that take each element from the product by order, so this creates exactly the effect you were looking for - whenever one of them frees up, it will take the next pair in line.

Upvotes: 1

Eric Truett
Eric Truett

Reputation: 3010

Combine the parameters in a list comprehension. Then you can use multiprocessing.Pool.starmap to call a function with multiple arguments.

from multiprocessing import Pool

x1 = ['a','b','c','d','e']
x2 = [1,2,3]

param_combinations = [(i, j) for i in x1 for j in x2] 

def func(x, y):
    return f'{x}_{y}'

with Pool(processes=3) as p:
    results = p.starmap(func, param_combinations)

>>>results
['a_1',
 'a_2',
 'a_3',
 'b_1',
 'b_2',
 'b_3',
 'c_1',
 'c_2',
 'c_3',
 'd_1',
 'd_2',
 'd_3',
 'e_1',
 'e_2',
 'e_3']

Upvotes: 1

Related Questions