CODEWITHSUNDEEP
pythonasynchronoustornadogevent
user1496984
user1496984

Reputation: 11575

gevent to Tornado ioloop - Structure code with coroutines/generators

I'm trying to convert some fairly straightforward gevent code to use the async facilities of Tornado. The sample code below uses the ZMQ library to do a very simple request-response.

import zmq.green as zmq

def fun():
    i = zmq.Context.instance()
    sock = i.socket(zmq.REQ)
    sock.connect('tcp://localhost:9005')
    sock.send('Ping')
    return sock.recv()

I can run this as fun() anywhere in my code. The .recv() call blocks while waiting for a reply, and the gevent hub can schedule the other parts of the code. When values are received, the function returns the value.

I read the problems that can arise with these implicit returns, and I want to run this using the Tornado IOLoop (also because I want to run it within the IPython Notebook). The following is an option, where recv_future() returns a Future that contains the result:

@gen.coroutine
def fun():
    i = zmq.Context.instance()
    sock = i.socket(zmq.REQ)
    sock.connect('tcp://localhost:9005')
    sock.send('Ping')
    msg = yield recv_future(sock)
    print "Received {}".format(msg[0])
    raise gen.Return(msg)

def recv_future(socket):
    zmqstream = ZMQStream(socket)  # Required for ZMQ
    future = Future()
    def _finish(reply):
        future.set_result(reply)
    zmqstream.on_recv(_finish)
    return future

The problem is that now fun() is not a function, but is a generator. So if I need to call it from another function, I need to use yield fun(). But then the calling function also becomes a generator!

What is the right way to structure code that uses Python generators? Do I have to make every function a generator to make it work? What if I need to call one of these functions from __init__()? Should that also become a generator?

Upvotes: 4

Views: 687

Answers (1)

dano
dano

Reputation: 94881

What if I need to call one of these functions from __init__()? Should that also become a generator?

This is one of the currently unsolved issues with explicit asynchronous programming with yield /yield from (on Python 3.3+). Magic methods don't support them. You can read some interesting thoughts from a Python core developer on asynchronous programming that touches on this issue here.

What is the right way to structure code that uses Python generators? Do I have to make every function a generator to make it work? Not every function, but every function that you want to call a coroutine, and wait for that coroutine to finish before continuing. When you switch to an explicit asynchronous programming model, you generally want to go all-in with it - your entire program runs inside the tornado ioloop. So, with this toy example, you would just do:

from tornado.ioloop import IOLoop
from tornado.gen import coroutine
from tornado.concurrent import Future

@gen.coroutine
def fun():
    i = zmq.Context.instance()
    sock = i.socket(zmq.REQ)
    sock.connect('tcp://localhost:9005')
    sock.send('Ping')
    msg = yield recv_future(sock)
    print "Received {}".format(msg[0])
    raise gen.Return(msg)

def recv_future(socket):
    zmqstream = ZMQStream(socket)  # Required for ZMQ
    future = Future()
    def _finish(reply):
        future.set_result(reply)
    zmqstream.on_recv(_finish)
    return future

if __name__ == "__main__":
    ioloop = IOLoop.instance()
    ioloop.add_callback(fun)
    ioloop.start() # This will run fun, and then block forever.
    #ioloop.run_sync(fun) # This will start the ioloop, run fun, then stop the ioloop

It looks like you might be able to get access to the ioloop IPython is using via the IPython.kernel API:

In [4]: from IPython.kernel.ioloop import manager

In [5]: manager.ioloop.IOLoop.instance()
Out[5]: <zmq.eventloop.ioloop.ZMQIOLoop at 0x4249ac8>

Upvotes: 1

Related Questions