CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessing
cjp94
cjp94

Reputation: 53

Exception handling in Pool callback

Given this example scenario:

def _callback(result):
    if result == 2:
        # introduce an exception into one of the callbacks
        raise Exception("foo")

    print (result)

def _target(v):
    return v

worker_pool = Pool()

for i in range(10):
    worker_pool.apply_async(_target, args=(i,), callback=_callback)

worker_pool.close()
worker_pool.join()

I was hoping to see each value of i printed except for i=2, which would instead have yielded an exception.

Instead I see something like the following:

0
1
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 479, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 649, in _set
    self._callback(self._value)
  File "test3.py", line 6, in _callback
    raise Exception("foo")
Exception: foo

... and then execution just hangs.

I'm aware that Pool handles callbacks on a separate thread, but why would execution hang and how can I reliably guard against errors in a task's callback?

Upvotes: 0

Views: 901

Answers (1)

Gustavo Kawamoto
Gustavo Kawamoto

Reputation: 3067

This is happening because the exception inside the callback method is basically killing the thread that handles the Pool, as it does not have an except block to handle this kind of situation. After the Thread is dead, it's unable to join the worker_pool, so your application hangs.

I believe that's a decision made by the Python maintainers, so the best way to handle this exception is to envelop your code inside a try/except block and handle it, instead of bubbling and letting the thread be killed.

Upvotes: 1

Related Questions