Using a context manager with a coroutine

Question

This code does not work

from contextlib import contextmanager                                                     
import tornado.ioloop                                                                     
import tornado.web                                                                        
from tornado import gen                                                                   
from tornado.httpclient import AsyncHTTPClient                                            


@contextmanager                                                                           
def hello():                                                                              
    print("hello in")                                                                     
    yield                                                                                 
    print("hello out")                                                                    


class MainHandler(tornado.web.RequestHandler):                                            
    @gen.coroutine                                                                        
    def get(self):                                                                        
        client = AsyncHTTPClient()                                                        
        with hello():                                                                     
            result = yield client.fetch("http://localhost")                               
        return "Hello "+str(result)                                                       

app = tornado.web.Application([('/', MainHandler)])                                       
app.listen(12345)                                                                         
tornado.ioloop.IOLoop.current().start()                                                   

And the reason why it doesn't work is that the context manager yielding and the coroutine yielding are incompatible in their behavior.

Do you confirm that the only way to achieve this is to use a try finally (particularly annoying if the context manager code must be used in many places). Maybe there's a subtle trick I don't know about? Googling did not help.

edit

This is the output I get

(venv) sborini@Mac-34363bd19f52:tornado$ python test.py 
hello in
ERROR:tornado.application:Uncaught exception GET / (::1)
HTTPServerRequest(protocol='http', host='localhost:12345', method='GET', uri='/', version='HTTP/1.1', remote_ip='::1', headers={'Upgrade-Insecure-Requests': '1', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36', 'Accept-Language': 'en-US,en;q=0.8,it;q=0.6', 'Connection': 'keep-alive', 'Host': 'localhost:12345', 'Accept-Encoding': 'gzip, deflate, sdch'})
Traceback (most recent call last):
  File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/web.py", line 1445, in _execute
    result = yield result
  File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "test.py", line 20, in get
    result = yield client.fetch("http://localhost")
  File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/Users/sborini/Work/Experiments/tornado/venv/lib/python3.4/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
ConnectionRefusedError: [Errno 61] Connection refused
ERROR:tornado.access:500 GET / (::1) 5.04ms

The point is that I never get the hello out message. I would expect that, once fetch spawns the future and the future errors, I return back to the yield point, get the exception, and leave the context, triggering the print('hello out').

Note that I do get hello out if I just do a try: finally: around the yield

Answer 1

This is a mechanic of contextlib.contextmanager, when an exception is raised inside the with block that error is thrown into hello so that it has access to the exception details and is given a change to suppress it (like a true context manager) for example:

from contextlib import contextmanager                                                     

@contextmanager                                                                           
def hello():                                                                              
    print("hello in")
    try:
        yield
    except:
        print("an exception was thrown into the generator! exit code would not have been run!")
        raise #commenting this out would suppress the original error which __exit__ can do by returning True
    finally:
        print("hello out")                                                                    


def get():                                                       
    with hello():                                                                     
        result = yield "VALUE"                              
    return "Hello "+str(result)                                                       

gen = get()
next(gen)
gen.throw(TypeError)

the output of this code example is:

hello in
an exception was thrown into the generator! exit code would not have been run!
hello out
Traceback (most recent call last):
  File "/Users/Tadhg/Documents/codes/test.py", line 24, in <module>
    gen.throw(TypeError)
  File "/Users/Tadhg/Documents/codes/test.py", line 19, in get
    result = yield "VALUE"
TypeError

For this reason I'd recommend using a simple context class instead of using contextlib.contextmanager since the scematics will be easier with explicit __enter__ and __exit__:

class hello:
    def __enter__(self):
        print("hello in")
    def __exit__(self,*args):
        print("hello out")

This way you are guaranteed that the exit code will run at the end of the with block.

Answer 2

The structure of the code is correct, and it's fine to mix context managers and coroutines this way. The @contextmanager and @coroutine decorators each assign their own meanings to yield within their decorated functions, but they remain independent.

As written, this code will print "hello in" and "hello out" if the fetch to http://localhost succeeds (or if you change it to point to a server that works), but it won't print "hello out" if the fetch raises an exception. To do that, you need to use a try/finally in your decorator:

@contextmanager
def hello():                                                                              
    print("hello in")                                                                     
    try:
        yield                                                                                 
    finally:
        print("hello out")   

One other error in this code is that you're returning a value from get(). The return value of get() is ignored; in Tornado to produce output you must call self.write() (or finish() or render()).