CODEWITHSUNDEEP
pythonexceptionpython-multiprocessing
akshan
akshan

Reputation: 363

Python Multiprocessing Doesnt Terminate On Base Exception

When running using multiprocessing pool, I find that the worker process keeps running past a point where an exception is thrown.

Consider the following code:

import multiprocessing


def worker(x):
    print("input: " + x)
    y = x + "_output"
    raise Exception("foobar")
    print("output: " + y)
    return(y)


def main():

    data = [str(x) for x in range(4)]
    pool = multiprocessing.Pool(1)
    chunksize = 1
    results = pool.map(worker, data, chunksize)
    pool.close()
    pool.join()

    print("Printing results:")
    print(results)


if __name__ == "__main__":
    main()

The output is:

$ python multiprocessing_fail.py
input: 0
input: 1
input: 2
Traceback (most recent call last):
input: 3
  File "multiprocessing_fail.py", line 25, in <module>
    main()
  File "multiprocessing_fail.py", line 16, in main
    results = pool.map(worker, data, 1)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
Exception: foobar

As you can see, the worker process never proceeds beyond raise Exception("foobar") to the second print statement. However, it resumes work at the beginning of function worker() again and again.

I looked for an explanation in the documentation, but couldn't find any. Here is a potentially related SO question:

Keyboard Interrupts with python's multiprocessing Pool

But that is different (about keyboard interrupts not being picked by the master process).

Another SO question:

How to catch exceptions in workers in Multiprocessing

This question is also different, since in it the master process doesnt catch any exception, whereas here the master did catch the exception (line 16). More importantly, in that question the worker did not run past an exception (there is only one executable line for the worker).

Am running python 2.7

Upvotes: 2

Views: 1409

Answers (1)

stovfl
stovfl

Reputation: 15513

Comment: Pool should start one worker since the code has pool = multiprocessing.Pool(1).

From the Documnentation:
A process pool object which controls a pool of worker processes to which jobs can be submitted

Comment: That one worker is running the worker() function multiple times

From the Documentation:
map(func, iterable[, chunksize])
This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks.

Your worker() is the separate task. Renaming your worker() to task() could help to clarify what is what.

Comment: What I expect is that the worker process crashes at the Exception

It does, the separate task, your worker() dies and Pool starts the next task.

What you want is Pool.terminate()

From the Documentation:

terminate()
Stops the worker processes immediately without completing outstanding work.

Question: ... I find that the worker process keeps running past a point where an exception is thrown.

You give iteration data to Pool, therfore Pool does what it have to do:
Starting len(data) worker.

data = [str(x) for x in range(4)]

The main Question is: What do you want to expect with 

raise Exception("foobar")

Upvotes: 1

Related Questions