CODEWITHSUNDEEP
pythonasynchronousexceptiontornadocoroutine
razvan.paul.blaga
razvan.paul.blaga

Reputation: 43

Python tornado parallel requests using coroutines and handling exceptions

I've started working with python tornado recently.

I've done a small project using callbacks and now I'm trying to figure out how to use coroutines to make my life a bit easier ( or harder )

I have the following code:

from tornado import gen

import random
import time

class App(object):

    @gen.coroutine
    def get_repo_issues(self, repo):
        sleep_time = random.randint(1, 5)
        time.sleep(sleep_time)

        if sleep_time in (2, 4):
            raise Exception('There is a disturbance in the force... %d' % sleep_time)

        raise gen.Return(sleep_time)


    @gen.coroutine
    def get_all_repo_issues(self, repo_list):

        for i in repo_list:
            repo = 'repo_' + str(i)
            try:
                items = yield self.get_repo_issues(repo)
                print 'Got %d items for %s' % (items, repo)
            except Exception as e:
                print 'Got exception for %s: %s' % (repo, str(e))


    @gen.coroutine
    def main(self):
        yield gen.Task(self.get_all_repo_issues, range(5))


def main():
    App().main()

Above code works as intended but I want parallel calls to self.get_repo_issues while catching and treating exceptions.

I was thinking something along the lines of yielding a dictionary of calls but dont know how to catch exceptions.

Is there a way to do this in tornado ?

Upvotes: 2

Views: 2033

Answers (2)

Spouk
Spouk

Reputation: 712

For blocking functions use tornado way - http://tornado.readthedocs.org/en/latest/guide/coroutines.html#calling-blocking-functions

Upvotes: 0

A. Jesse Jiryu Davis
A. Jesse Jiryu Davis

Reputation: 24007

Calling sleep in a Tornado application is forbidden. Tornado apps are usually single-threaded, so sleep blocks the entire process and no other coroutine or callback can run while sleep is executing. Test with yield gen.sleep(n_seconds) instead.

To wait for many operations, something like this (untested) using WaitIterator:

futures = {}
for i in repo_list:
    repo = 'repo_' + str(i)
    futures[repo] = self.get_repo_issues(repo)

wait_iterator = gen.WaitIterator(**futures)
while not wait_iterator.done():
    try:
        items = yield wait_iterator.next()
    except Exception as e:
        print 'Got exception for %s: %s' % (
            wait_iterator.current_index, str(e))
    else:
        repo = wait_iterator.current_index
        print 'Got %d items for %s' % (items, repo)

Upvotes: 3

Related Questions