CODEWITHSUNDEEP
pythonasynchronoustornadodecoratorcoroutine
zijuexiansheng
zijuexiansheng

Reputation: 347

Why does the order of asynchronous and gen.coroutine matter in Tornado?

I have a piece of code as follows:

@tornado.web.stream_request_body
class DownloadHandler(SecureHandler):
    executor = ThreadPoolExecutor(50)

    @tornado.web.authenticated
    @tornado.gen.coroutine
    @tornado.gen.asynchronous
    def post(self):
        # ...

        path = yield self.down_load(fname) 

        self.set_header("Content-Type", "application/octet-stream")
        self.set_header("Content-Disposition", "attachment;filename=%s" % fname)

        self.generator = self.read_file(path)
        tornado.ioloop.IOLoop.instance().add_callback(self.loop)

    @run_on_executor
    def down_load(self, fname):
        # download a file named `fname` from other website
        # store it in a temp file at `path`
        # ...
        return path

    def loop(self):
        try:
            data = self.generator.next()
            self.write(data)
            self.flush()
            tornado.ioloop.IOLoop.instance().add_callback(self.loop)
        except Exception as e:
            traceback.print_exc()
            self.finish()

    def read_file(self, fname):
        with open(fname, 'rb') as f:
            while True:
                data = f.read(1024 * 1024 * 8)
                if not data
                    break
                yield data

If the order of asynchronous and gen.coroutine is as shown in my code, it works fine.

But if I switch the order of them, the client side can only receive 8MB data. the traceback.print_exc() prints finish() called twice. May be caused by using async operations without the @asynchronous decorator.

So my question, why does the order of these two decorators matter, and what are the rules to choose the order?

Upvotes: 2

Views: 1242

Answers (1)

Ben Darnell
Ben Darnell

Reputation: 22154

Order matters because @asynchronous looks at the Future returned by @gen.coroutine, and calls finish for you when the coroutine returns. Since Tornado 3.1, the combination of @asynchronous and @gen.coroutine has been unnecessary and discouraged; in most cases you should use @gen.coroutine alone.

However, the example you've shown here is kind of odd - it mixes the coroutine and callback style in ways that don't really work well. The coroutine returns before it is finished, and leaves the remaining work to a chain of callbacks. This is actually similar in spirit to what the @asynchronous decorator does with non-coroutine functions, although it doesn't work with the interaction between the two decorators. The best solution here is to make loop() a coroutine too:

@gen.coroutine
def loop(self):
    for data in self.generator:
        self.write(data)
        yield self.flush()

Then you can call it with yield self.loop() and the coroutine will work normally. You no longer need to call finish() explicitly, or use the @asynchronous decorator.

Upvotes: 2

Related Questions