CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
vin
vin

Reputation: 1019

Multiprocessing Pool non-blocking detection of finished tasks

Is there a way know that a process pool has finished running the tasks?

_process_pool = mp.Pool(processes=num_workers)

I am adding task onto the pool by batches:

for batch in gen_batches():
   _process_pool.map_async(fn, batch)

Is there a way to know when all tasks have been done? callback doesn't seem to work here. And I want to avoid to block the parent process by calling _process_pool.join()

Upvotes: 4

Views: 3331

Answers (1)

stovfl
stovfl

Reputation: 15533

Question: Is there a way to know when all tasks have been done?

Append all AsyncResult from pool.map_async(... to a list, for instance:

multiple_results = []
for batch in gen_batches():
   multiple_results.append( _process_pool.map_async(fn, batch) )


if all([ar.ready() for ar in multiple_results]):
    print('Pool done')
else:
    print('Pool any alive')

Python » 3.6.1 Documentation: multiprocessing.pool.AsyncResult

Tested with Python:3.4.2

Upvotes: 4

Related Questions