CODEWITHSUNDEEP
pythonpython-3.xexceptionmultiprocessingpython-multiprocessing
Valery Kustov
Valery Kustov

Reputation: 105

How to pass stacktrace between processes in Python?

I'm trying to create a python decorator which takes a function with args and kwargs, executes it in a new process, shuts it down and returns whatever the function returned, including raising the same exception, if any.

For now, my decorator handles functions okay, if they raise no exceptions, but fails to provide the traceback. How do I pass it back to the parent process?

from functools import wraps
from multiprocessing import Process, Queue
import sys


def process_wrapper(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        # queue for communicating between parent and child processes
        q = Queue()

        def func_to_q(_q: Queue, *_args, **_kwargs):
            # do the same as func, but put result into the queue. Also put
            # there an exception if any.
            try:
                _res = func(*_args, **_kwargs)
                _q.put(_res)
            except:
                _q.put(sys.exc_info())

        # start another process and wait for it to join
        p = Process(target=func_to_q, args=(q, )+args, kwargs=kwargs)
        p.start()
        p.join()

        # get result from the queue and return it, or raise if it's an exception
        res = q.get(False)
        if isinstance(res, tuple) and isinstance(res[0], Exception):
            raise res[1].with_traceback(res[2])
        else:
            return res
    return wrapper


if __name__ == '__main__':

    @process_wrapper
    def ok():
        return 'ok'

    @process_wrapper
    def trouble():
        def inside():
            raise UserWarning
        inside()

    print(ok())
    print(trouble())

I expect result to be something like:

ok
Traceback (most recent call last):
  File "/temp.py", line 47, in <module>
    print(trouble())
  File "/temp.py", line 44, in trouble
    inside()
  File "/temp.py", line 43, in inside
    raise UserWarning
UserWarning

Process finished with exit code 1

But it seems like the child process cannot put stacktrace into the queue and I get the following:

ok
Traceback (most recent call last):
  File "/temp.py", line 47, in <module>
    print(trouble())
  File "/temp.py", line 26, in wrapper
    res = q.get(False)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 107, in get
    raise Empty
queue.Empty

Process finished with exit code 1

Also, if the child puts into the queue only the exception itself _q.put(sys.exc_info()[1]), parent gets it from there and raises but with new stacktrace (note missing call to inside()):

ok
Traceback (most recent call last):
  File "/temp.py", line 47, in <module>
    print(trouble())
  File "/temp.py", line 28, in wrapper
    raise res
UserWarning

Process finished with exit code 1

Upvotes: 3

Views: 317

Answers (1)

Darkonaut
Darkonaut

Reputation: 21654

Take a look at multiprocessing.pool.py and the stringification-hack for sending Exceptions to the parent. You can use multiprocessing.pool.ExceptionWithTraceback from there.

That's just enough code for demonstrating the basic principle:

from multiprocessing import Process, Queue
from multiprocessing.pool import ExceptionWithTraceback


def worker(outqueue):
    try:
        result = (True, 1 / 0)  # will raise ZeroDivisionError
    except Exception as e:
        e = ExceptionWithTraceback(e, e.__traceback__)
        result = (False, e)
    outqueue.put(result)

if __name__ == '__main__':

    q = Queue()
    p = Process(target=worker, args=(q,))
    p.start()
    success, value = q.get()
    p.join()

    if success:
        print(value)
    else:
        raise value  # raise again

Output:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/...", line 7, in worker
    result = (True, 1 / 0)  # will raise ZeroDivisionError
ZeroDivisionError: division by zero
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/...", line 23, in <module>
    raise value
ZeroDivisionError: division by zero

Process finished with exit code 1

Upvotes: 4

Related Questions