CODEWITHSUNDEEP
pythonparallel-processingmultiprocessingpoolpython-multithreading
mnowotka
mnowotka

Reputation: 17228

Python multiprocessing pool for parallel processing

I want to execute some processes in parallel and wait until they finish. So I wrote this code:

pool = mp.Pool(5)
for a in table:
    pool.apply(func, args = (some_args))
pool.close()
pool.join()

Will I get 5 processes executing func in parallel here? Or the only option is apply_async?

Upvotes: 5

Views: 8121

Answers (2)

johntellsall
johntellsall

Reputation: 15170

Another solution is to use Pool.imap_unordered()

The following code starts a pool of 5 workers. It then sends three jobs to the pool. The first one is num=1, second num=2, etc. The function imap_unordered means that when the first result appears, from any worker, return it for further processing. In this case, the loop prints results as they appear, which isn't in any specific order.

import multiprocessing

def calc(num):
    return num*2

pool = multiprocessing.Pool(5)
for output in pool.imap_unordered(calc, [1,2,3]):
    print 'output:',output

Upvotes: 3

Janne Karila
Janne Karila

Reputation: 25197

The docs are quite clear on this: each call to apply blocks until the result is ready. Use apply_async.

Upvotes: 5

Related Questions