CODEWITHSUNDEEP
pythonpython-2.7multiprocessing
Dschoni
Dschoni

Reputation: 3862

How to catch exceptions in workers in Multiprocessing

I'm working with the multiprocessing module in Python (2.7.3) and want to debug some stuff going on in my workers. However, I seem to not be able to catch any exceptions in the worker threads.

A minimal example:

import multiprocessing as mp


a=[1]
def worker():
    print a[2]


def pool():
    pool = mp.Pool(processes=1)
    pool.apply_async(worker, args = ())
    pool.close()
    pool.join()
    print "Multiprocessing done!"


if __name__ == '__main__':
    pool()

This is expected to raise an IndexError, but my output only is 

    Multiprocessing done!

Is there a way to show me all exceptions occuring in the worker threads without manually raising my own?

Upvotes: 18

Views: 18220

Answers (2)

falsetru
falsetru

Reputation: 369064

The error is not raised unless you call get method of AsyncResult (the return value of the apply_async):

According to the AsyncResult.get documentation:

Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get().

def pool():
    pool = mp.Pool(processes=1)
    result = pool.apply_async(worker, args=())
    result.get() # <------------
    pool.close()
    pool.join()
    print "Multiprocessing done!"

Upvotes: 32

KobeJohn
KobeJohn

Reputation: 7545

I think falsetru gave you what you need. I'd just like to expand a little more.

If it's important for you to get not only the error but the original context (i.e. to know that the exception occurred on line 1 of worker()) then you can check this nice post by Ned Batchelder which explains how to reraise exceptions with their original context.

That doesn't work for mp.Pool so it's just in case you need something more. This SO Question covers your question using more explicit multiprocessing techniques instead of mp.Pool.

Upvotes: 3

Related Questions