How to execute Tornado coroutine inside of synchronous environment?

Question

I have some Tornado's coroutine related problem.

There is some python-model A, which have the abbility to execute some function. The function could be set from outside of the model. I can't change the model itself, but I can pass any function I want. I'm trying to teach it to work with Tornado's ioloop through my function, but I couldn't.

Here is the snippet:

import functools
import pprint
from tornado import gen
from tornado import ioloop

class A:
    f = None
    def execute(self):
      return self.f()
    pass

@gen.coroutine
def genlist():
    raise gen.Return(range(1, 10))

@gen.coroutine
def some_work():
    a = A()
    a.f = functools.partial(
        ioloop.IOLoop.instance().run_sync,
        lambda: genlist())
    print "a.f set"
    raise gen.Return(a)

@gen.coroutine
def main():
    a = yield some_work()
    retval = a.execute()
    raise gen.Return(retval)


if __name__ == "__main__":
    pprint.pprint(ioloop.IOLoop.current().run_sync(main))

So the thing is that I set the function in one part of code, but execute it in the other part with the method of the model.

Now, Tornado 4.2.1 gave me "IOLoop is already running" but in Tornado 3.1.1 it works (but I don't know how exactly).

I know next things:

1. I can create new ioloop but I would like to use existent ioloop.
2. I can wrap genlist with some function which knows that genlist's result is Future, but I don't know, how to block execution until future's result will be set inside of synchronous function.

Also, I can't use result of a.execute() as an future object because a.execute() could be called from other parts of the code, i.e. it should return list instance.

So, my question is: is there any opportunity to execute asynchronous genlist from the synchronous model's method using current IOLoop?

Answer 1

Say, your function looks something like this:

@gen.coroutine
def foo():
    # does slow things

or

@concurrent.run_on_executor
def bar(i=1):
    # does slow things

You can run foo() like so:

from tornado.ioloop import IOLoop
loop = IOLoop.current()

loop.run_sync(foo)

You can run bar(..), or any coroutine that takes args, like so:

from functools import partial
from tornado.ioloop import IOLoop

loop = IOLoop.current()

f = partial(bar, i=100)
loop.run_sync(f)

Answer 2

You cannot restart the outer IOLoop here. You have three options:

1. Use asynchronous interfaces everywhere: change a.execute() and everything up to the top of the stack into coroutines. This is the usual pattern for Tornado-based applications; trying to straddle the synchronous and asynchronous worlds is difficult and it's better to stay on one side or the other.
2. Use run_sync() on a temporary IOLoop. This is what Tornado's synchronous tornado.httpclient.HTTPClient does, which makes it safe to call from within another IOLoop. However, if you do it this way the outer IOLoop remains blocked, so you have gained nothing by making genlist asynchronous.

3. Run a.execute on a separate thread and call back to the main IOLoop's thread for the inner function. If a.execute cannot be made asynchronous, this is the only way to avoid blocking the IOLoop while it is running.

   executor = concurrent.futures.ThreadPoolExecutor(8)
   
   @gen.coroutine
   def some_work():
       a = A()
       def adapter():
           # Convert the thread-unsafe tornado.concurrent.Future
           # to a thread-safe concurrent.futures.Future.
           # Note that everything including chain_future must happen
           # on the IOLoop thread.
           future = concurrent.futures.Future()
           ioloop.IOLoop.instance().add_callback(
               lambda: tornado.concurrent.chain_future(
                   genlist(), future)
           return future.result()
       a.f = adapter
       print "a.f set"
       raise gen.Return(a)
   
   @gen.coroutine
   def main():
       a = yield some_work()
       retval = yield executor.submit(a.execute)
       raise gen.Return(retval)