What should I decorate with @asyncio.coroutine for async operations?

Question

I want to run my code asnychronously. What should I decorate with @asyncio.coroutine and what should I call with yield from for async operations?

In my case, I have some example code without decorator. (Simple chat bot look-like IRC)

import asyncio


class ChatBot:
    def __init__(self, loop):
        conn = asyncio.open_connection(HOST, PORT, loop=loop)
        self.reader, self.writer = yield from conn

    def send(self, msg):
        self.writer.write(msg)

    def read(self):
        msg = yield from self.reader.readline()
        return msg

    def run(self):
        while True:
            msg = self.read()
            self.parse(msg)

    def parse(self, msg):
        if msg.startswith('PING'):
            self.some_work(msg)
        elif msg.startswith('ERROR'):
            self.some_error()
        else:
            self.server_log(msg)

    def some_work(self, msg):
        # some work. It can call asynchronous function like I/O or long operation. It can use self.send().

    def some_error(self, msg):
        # some work. It can call asynchronous function like I/O or long operation. It can use self.send().

    def server_log(self, msg):
        # some work. It can call asynchronous function like I/O or long operation. It can use self.send().


loop = asyncio.get_event_loop()
bot = ChatBot(loop)
loop.run_until_complete(???)
loop.close()

I think ??? is bot.run() and ChatBot.run must be decorated with @asyncio.coroutine. Then, how about other methods? I can't understand when use @asyncio.coroutine decorator and call method with yield from or asyncio.async. (I already read PEP-3156 for understanding asnycio. But I can't understand fully.)

Answer 1

When to use the @asyncio.coroutine decorator

If you have a function that needs to use yield from to call a coroutine, you should decorate it with asyncio.coroutine. Also note that coroutines are often (not always) "viral". As soon as you add yield from to a function it becomes a coroutine, and additionally any function that calls that coroutine usually (though not always) needs to be come a coroutine, too.

When to use asyncio.async

Why are coroutines not always viral? Because you actually don't always need to use yield from to call a coroutine. You only need to use yield from if you want to call a coroutine and wait for it to finish. If you just want to kick off a coroutine in the background, you can just do this:

asyncio.async(coroutine())

This will schedule coroutine to run as soon as control returns to the event loop; it won't wait for coroutine to finish before moving on to the next line. An ordinary function can use this to schedule a coroutine to run without also having to become a coroutine itself.

You can also use this approach to run multiple coroutines concurrently. So, imagine you have these two coroutines:

@asyncio.coroutine
def coro1():
   yield from asyncio.sleep(1)
   print("coro1")

@asyncio.coroutine
def coro2():
   yield from asyncio.sleep(2)
   print("coro2")

If you had this:

@asyncio.coroutine
def main():
    yield from coro1()
    yield from coro2()
    yield from asyncio.sleep(5)

asyncio.get_event_loop().run_until_complete(main())

After 1 second, "coro1" would be printed. Then, after two more seconds (so three seconds total), "coro2" would be printed, and five seconds later the program would exit, making for 8 seconds of total runtime. Alternatively, if you used asyncio.async:

@asyncio.coroutine
def main():
    asyncio.async(coro1())
    asyncio.async(coro2())
    yield from asyncio.sleep(5)


asyncio.get_event_loop().run_until_complete(main())

This will print "coro1" after one second, "coro2" one second later, and the program would exit 3 seconds later, for a total of 5 seconds of runtime.

How does this affect your code?

So following those rules, your code needs to look like this:

import asyncio

class ChatBot:
    def __init__(self, reader, writer):
        # __init__ shouldn't be a coroutine, otherwise you won't be able
        # to instantiate ChatBot properly. So I've removed the code that
        # used yield from, and moved it outside of __init__.
        #conn = asyncio.open_connection(HOST, PORT, loop=loop)
        #self.reader, self.writer = yield from conn
        self.reader, self.writer = reader, writer

    def send(self, msg):
        # writer.write is not a coroutine, so you 
        # don't use 'yield from', and send itself doesn't 
        # need to be a coroutine.
        self.writer.write(msg)

    @asyncio.coroutine
    def read(self):
        msg = yield from self.reader.readline()
        return msg

    @asyncio.coroutine
    def run(self):
        while True:
            msg = yield from self.read()
            yield from self.parse(msg)

    @asyncio.coroutine
    def parse(self, msg):
        if msg.startswith('PING'):
            yield from self.some_work(msg)
        elif msg.startswith('ERROR'):
            yield from self.some_error()
        else:
            yield from self.server_log(msg)

    @asyncio.coroutine
    def some_work(self, msg):
        # some work. It can call asynchronous function like I/O or long operation. It can use self.send().

    @asyncio.coroutine
    def some_error(self, msg):
        # some work. It can call asynchronous function like I/O or long operation. It can use self.send().

    @asyncio.coroutine
    def server_log(self, msg):
        # some work. It can call asynchronous function like I/O or long operation. It can use self.send()

@asyncio.coroutine
def main(host, port):
    reader, writer = yield from asyncio.open_connection(HOST, PORT, loop=loop)
    bot = ChatBot(reader, writer)
    yield from bot.run()


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

One other thing to keep in mind - adding yield from in front of a function doesn't magically make that call non-blocking. Neither does adding the @asyncio.coroutine decorator. Functions are only non-blocking if they're actually directly or indirectly calling native asyncio coroutines, which use non-blocking I/O and are integrated with the asyncio event loop. You mentioned making REST API calls, for example. In order for those REST API calls to not block the event loop, you'd need to use the aiohttp library, or asyncio.open_connection. Using something like requests or urllib will block the loop, because they're not integrated with `asyncio.

Answer 2

You should decorate everything that uses yield from, because the decorator asyncio.coroutine will take your function as a generator and do all the callback/async job while you're yielding.

In your case, run needs to be rewritten as the following:

@asyncio.coroutine
def run(self):
    while True:
        msg = yield from self.read()
        yield from self.parse(msg)

Then, read and parse must be coroutine as well. You should read about how asynchronous work before use it, it will help you a lot.

Answer 3

@asyncio.coroutine
def read(self):
    msg = yield from self.reader.readline()
    return msg

@asyncio.coroutine
def run(loop):
    while True:
        msg = yield from read()
        yield from parse(msg)

loop = asyncio.get_event_loop()
loop.run_until_complete(run(loop))
loop.close()