CODEWITHSUNDEEP
python-3.xmultiprocessing
sobmortin
sobmortin

Reputation: 55

Wait for all processes to finish python

This question has been asked numerous times, but I couldn't find an answer to match my specific set up.

I have a script which uses multiprocessing. Here is a much simplified example:

from multiprocessing import Pool

def my_function(arg1,arg2):
     print(arg1,arg2)

with Pool(processes=5) as pool:
     pool.starmap(function, [(arg1,arg2),(arg1,arg2)])
     pool.join()

print("should only run once all processes are finished")

I would like the pool to close at this point and normal single processing functionality to continue. I thought the context manager would handle the closure of the pool, and the .join() would ensure that it only closes once all processes are finished. But I am seeing behaviour which suggests that the program is running beyond the context manager before all processes are finished. Is there more I need to do at this point?

Also, in this case there are 5 processes assigned, but only 2 tuples of args passed to the function. What do the surplus 3 processes do in this instance?

Python -V 3.9

Upvotes: 1

Views: 2538

Answers (1)

Dariyoush
Dariyoush

Reputation: 508

You need to call pool.close() before pool.join(). and you can see which process running the function, by calling _identity variable of a current process. In the end, you can make sure that only the main process is running by calling the name of the current process. since the number of calls is less than the number of processes, the rest of the processes are just doing the usual OS stuff.


from multiprocessing import Pool, current_process
def my_function(arg1,arg2):
    rank = f'running on process: {current_process()._identity[0]}'
    print(rank, arg1,arg2)


arg1 = 'arg1'
arg2 = 'arg2'
with Pool(processes=5) as pool:
    pool.starmap(my_function, [(arg1,arg2),(arg1,arg2)])
    pool.close()
    pool.join()

print(current_process().name)
print("should only run once all processes are finished")

output:

running on process: 1 arg1 arg2
running on process: 2 arg1 arg2
MainProcess
should only run once all processes are finished

Upvotes: 2

Related Questions