CODEWITHSUNDEEP
pythonloopsmultiprocessing
Ana
Ana

Reputation: 1586

running a function in parallel in Python

I'm trying to run a function like f in parallel in Python but have two problems:

  1. When using map, function f is not applied to all permuted tuples of arrays a and b.
  2. When trying to use Pool, I get the following error:
TypeError: '<=' not supported between instances of 'tuple' and 'int'

def f(n,m):
    x = n * m
    return x

a = (1,2,3,4,5)
b = (3,4,7,8,9)
result = map(f, a, b)
print(list(result))

#now trying parallel computing
from multiprocessing import Pool
pool = Pool(processes=4)
print(*pool.map(f, a, b))

Upvotes: 2

Views: 131

Answers (1)

Mark Tolonen
Mark Tolonen

Reputation: 177461

I didn't make any changes for your #1 issue and get the expected result from using map(). You seem to have an incorrect assumption of how it works, but didn't provide expectation vs. actual results for your example.

for #2 to return the same answers as #1, you need starmap() instead of map() for this instance of multiprocessing use, and then zip() the argument lists to provide sets of arguments. If on an OS that doesn't fork (and for portability if you are), run global code only if it is the main process, and not a spawned process by using the documented if __name__ == '__main__': idiom:

from multiprocessing import Pool

def f(n,m):
    x = n * m
    return x

if __name__ == '__main__':
    a = (1,2,3,4,5)
    b = (3,4,7,8,9)
    result = map(f, a, b)
    print(list(result))

    #now trying parallel computing
    pool = Pool(processes=4)
    print(*pool.starmap(f, zip(a, b)))

Output:

[3, 8, 21, 32, 45]
3 8 21 32 45

If you actually want permutations as mentioned in #1, use itertools.starmap or pool.starmap with itertools.product(a,b) as parameters instead.

Upvotes: 2

Related Questions