multiprocessing Pool killing threads unexpectedly

Question

I have a simple test program written and executed in python 3.6.3 below. It is being executed on a machine with 4 cores.

import multiprocessing
import time

def f(num):
  print(multiprocessing.current_process(), num)
  time.sleep(1)
  if (num % 2):
    raise Exception


pool = multiprocessing.Pool(5)

try:
  pool.map(f, range(1,20))
except Exception as e:
  print("EXCEPTION")

pool.close()
pool.join()

Output with pool = multiprocessing.Pool(5):

But if I change the process count of the pool to be equal to or less than the number of cores on my machine, each call to f() where num is even does not print.

output with pool = multiprocessing.Pool(4):

 1
 3
 5
 7
 9
 11
 13
 15
 17
 19
EXCEPTION

I don't understand why these processes are being killed, especially when the exception isn't even thrown until after the print statement in the function. I really don't understand why it only happens when the process count in the pool is equal to or less than the number of cores on the machine.

Valen · Accepted Answer

referring to the specification of multiprocessing.Pool.map you can see one optional argument chunksize, if you specify it to 1, i.e. pool.map(f, range(1,20), 1), then you would yield the expected result.

if you increase the chunk size (= 6 for example), you might see:

this suggests that number of chunksize of tasks are assigned to a single thread in the Pool, when you raise exception during each thread, of course the tasks in the remaining chuck would not be executed.

From here you can know that the default value for chunksize is 2 in your case, the reason of existence of such variable, to be seen fairly easily, is to reduce the number of new threads which need to be initialized (which might save both resources and processing time, when you have appropriate chunksize).

multiprocessing Pool killing threads unexpectedly

Answers (1)

Related Questions