CODEWITHSUNDEEP
pythonpython-multiprocessing
rgov
rgov

Reputation: 4329

Multiprocessing pool does not wait for job completion

import multiprocessing
import time

def job(i):
    print('I am job', i)
    while True:
        time.sleep(1)

with multiprocessing.Pool(processes=2) as pool:
    for i in range(2):
        pool.apply_async(job, args=(i,))
pool.join()

Shouldn't this wait forever for the two child processes to finish, which will never happen? Instead the parent process exits immediately.

This seems to be some consequence of using the context manager syntax; the following works:

pool = multiprocessing.Pool(processes=2)
for i in range(2):
    pool.apply_async(job, args=(i,))
pool.close()
pool.join()

Upvotes: 1

Views: 323

Answers (1)

Dartmoth89
Dartmoth89

Reputation: 31

Your original pool.join() is called outside of the with ... as context:

with multiprocessing.Pool(processes=2) as pool:
    for i in range(2):
        pool.apply_async(job, args=(i,))
pool.join()   # <-- indent is wrong

The correct version is:

with multiprocessing.Pool(processes=2) as pool:
    for i in range(2):
        pool.apply_async(job, args=(i,))
    pool.close()
    pool.join()

Upvotes: 2

Related Questions