CODEWITHSUNDEEP
pythonmultithreadingexceptiongevent
Phanto
Phanto

Reputation: 1167

ThreadPool exceptions in gevent do not get returned

I'm using gevent on Python 2.7 to do some multithreaded work. However, I am unable to catch exceptions that are raised in the spawned methods. I am using the .get() method on the returned AsyncResult object (returned when calling the .spawn() method on the thread pool).

Per the gevent documentation: http://www.gevent.org/gevent.event.html#gevent.event.AsyncResult.get:

get(block=True, timeout=None)

Return the stored value or raise the exception.

If this instance already holds a value or an exception, return or raise it immediately.

However, instead of returning the exception, .get() returns NoneType. Additionally, in the console, the exception details are printed out. Instead, .get() should return the Exception back.

Here is some sample code and corresponding output that showcases this:

from gevent import threadpool

def this_will_fail():
    raise Exception('This will fail')

result_list = []
for i in range(1,5):
    tpool = threadpool.ThreadPool(5)
    result_list.append(tpool.spawn(this_will_fail))

tpool.join()

for result in result_list:
    try:
        returned_object = result.get()
        print type(returned_object)

    except Exception as e:
        print 'This is not running for some reason...'

Output:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c79e50 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99210 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99410 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/threadpool.py", line 193, in _worker
    value = func(*args, **kwargs)
  File "<stdin>", line 2, in this_will_fail
Exception: This will fail
(<ThreadPool at 0x107c99610 0/1/5>, <function this_will_fail at 0x107c848c0>) failed with Exception

<type 'NoneType'>
<type 'NoneType'>
<type 'NoneType'>
<type 'NoneType'>

Am I missing something obvious, or is this a bug in gevent?

EDIT: concurrent.futures doesn't have this problem. While that is an acceptable solution for my use case, I would still like to understand why gevent doesn't properly return the exception.

Upvotes: 0

Views: 657

Answers (1)

Valentin
Valentin

Reputation: 26

Here we had the same problem with the gevent threadpool. We are using a solution which wrap the function to return exceptions as result, and raise it again when we want to use the result from the AsynResult.

class _ExceptionWrapper:
    def __init__(self, exception, error_string, tb):
        self.exception = exception
        self.error_string = error_string
        self.tb = tb

class wrap_errors(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        func = self.func
        try:
            return func(*args, **kwargs)
        except:
            return _ExceptionWrapper(*sys.exc_info())

    def __str__(self):
        return str(self.func)

    def __repr__(self):
        return repr(self.func)

    def __getattr__(self, item):
        return getattr(self.func, item)

def get_with_exception(g, block=True, timeout=None):
    result = g._get(block, timeout)
    if isinstance(result, _ExceptionWrapper):
        # raise the exception using the caller context
        raise result.error_string, None, result.tb
    else:
        return result

def spawn(fn, *args, **kwargs):
    # wrap the function
    fn = wrap_errors(fn)
    g = threadpool.spawn(fn, *args, **kwargs)
    # and the asynresult
    g._get = g.get
    g.get = types.MethodType(get_with_exception, g)
    return g

Upvotes: 1

Related Questions