CODEWITHSUNDEEP
pythondictionarymultiprocessing
Sergey Ivanov
Sergey Ivanov

Reputation: 3929

Multiprocessing map with multiple arguments

In view of randomized structure of my algorithm, I want to run it hundreds of times in parallel. Parameters stay the same every time.

Both of implementations raise some kind of exception (PicklingError, or TypeError).

from multiprocessing import Pool
from itertools import izip, repeat

if __name__ == '__main__':
    G = Class1Object()
    S = Class2Object()

    pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())

    # take 1 tuple input
    # parfun = lambda (G,S): fun(G,S)
    # T = pool.map(parfun, izip(repeat(G, 2), repeat(S, 2)))

    # take 2 arguments
    T = pool.map(fun, [G]*2, [S]*2)

What is the fastest and most convenient way to run function with multiple arguments in parallel?

Upvotes: 1

Views: 2755

Answers (1)

Multimedia Mike
Multimedia Mike

Reputation: 13216

The map() function only takes a single iterable. So perhaps you could create an iterable function that returns the different tuples you need (or just create an array of the tuples and pass that).

Be advised that you might have problems in trying to send objects across process boundaries. See my answer here in which I tackled the same issue.

Upvotes: 1

Related Questions