CODEWITHSUNDEEP
pythonexceptionasynchronoustornado
Cheikh Ahmadou
Cheikh Ahmadou

Reputation: 59

Catching exception while doing parallel async request with tornado

In my program, I am doing some get requests to some hosts. The issue is that I am unable to catch properly the exception that is thrown when a host is disconnected. I am using tornado and the request is asynchronous. Considering the code below:

   self.http_client = AsyncHTTPClient()

    try:
         responses = yield [self.http_client.fetch(theUrl) for theUrl in URLS]                                             
    except Exception as e:
         if (e[0] == 111) or (e[0] == 599):
               #Do something

When the host is disconnect sometimes I am able to catch the exception but it still get thrown. I get for instance this error message printed to my log files:

ERROR:tornado.application:Multiple exceptions in yield list
Traceback (most recent call last):
  File "/opt/felix-web-mon/env/lib/python2.7/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/opt/felix-web-mon/env/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
error: [Errno 111] Connection refused

Despite the fact that I am handling the '111' exception in my code, it is still being thrown. I suspect it is due to the fact that i am using list comprehension (which i need). How can I catch this 'Multiple exceptions in yield list' in an except block ? Could you help me ?

Upvotes: 1

Views: 1048

Answers (1)

kwarunek
kwarunek

Reputation: 12587

Awaiting for the multiple futures, by simply yielding a list, will "discard" all the responses on any error. You can either use:

  • WaiterIterator - the benefit of this one is that you gets result when it arrives. you're not waiting till all yield request are done (espaecially the slowest one).
  • pass raise_error=False in the fetch to suppress raising

Take a look at Exception handling for parallel fetch requests, both are described.

from tornado.gen import coroutine
from tornado.ioloop import IOLoop
from tornado.httpclient import AsyncHTTPClient

@coroutine
def main():
    URLS = [
        'http://amazon.com',
        'https://www.kernel.org/some404',
        'http://localhost:8787',  # connection refused
        'http://google.com'
    ]
    http_client = AsyncHTTPClient()

    responses = yield [http_client.fetch(theUrl, raise_error=False) for theUrl in URLS]
    for idx, r in enumerate(responses):
        print(URLS[idx])
        if 200 <= r.code <= 299:
            print('> ok')
        elif 300 <= r.code <= 399:
            print('> ok - redirect')
        elif 400 <= r.code <= 499:
            print('> client err: %s' % r.code)
        elif 500 <= r.code <= 598:
            print('> server err: %s' % r.code)
        elif r.code == 599:
            print('> 599 connection error or timedouted request')

        # or something like below
        #try:
        #    res = r.rethorw()
        #except Exception:
             # do something


IOLoop.instance().run_sync(main)

Upvotes: 1

Related Questions