Twisted webapp gets stuck when generating HTTP response

Question

I've created a web application using Twisted and SQLAlchemy. Since SQLAlchemy doesn't work together very well with Twisted's callback-based design (Twisted + SQLAlchemy and the best way to do it), I use deferToThread() within the root resource in order to run every request within its own thread. While this does generally work, about 10% of the requests get "stuck". This means that when I click a link in the browser, the request is handled by Twisted and the code for the respective resource runs and generates HTML output. But for whatever reason, that output is never sent back to the browser. Instead, Twisted sends the HTTP headers (along with the correct Content-Length), but never sends the body. The connection just stays open indefinitely with the browser showing the spinner icon. No errors are generated by Twisted in the logfile.

Below is a minimal example. If you want to run it, save it with a .tac extension, then run twistd -noy example.tac. On my server, the issue seems to occur relatively infrequently in this particular piece of code. Use something like while true; do wget -O- 'http://server.example.com:8080' >/dev/null; done to test it.

from twisted.web.server import Site
from twisted.application import service, internet
from twisted.web.resource import Resource
from twisted.internet import threads
from twisted.web.server import NOT_DONE_YET
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine, Column, Integer, String

Base = declarative_base()
class User(Base):
    '''A user account.'''
    __tablename__ = 'user'
    id = Column(Integer, primary_key=True)
    login = Column(String(64))


class WebInterface(Resource):

    def __init__(self):
        Resource.__init__(self)
        db_url = "mysql://user:password@mysql-server.example.com/myapp?charset=utf8"
        db_engine = create_engine(db_url, echo=False, pool_recycle=300) #discard connections after 300 seconds
        self.DBSession = sessionmaker(bind=db_engine)

    def on_request_done(self, _, request):
        '''All actions that need to be done after a request has been successfully handled.'''
        request.db_session.close()
        print('Session closed') #does get printed, so session should get closed properly


    def on_request_failed(self, err, call):
        '''What happens if the request failed on a network level, for example because the user aborted the request'''
        call.cancel()


    def on_error(self, err, request):
        '''What happens if an exception occurs during processing of the request'''
        request.setResponseCode(500)
        self.on_request_done(None, request)
        request.finish()
        return err 


    def getChild(self, name, request):
        '''We dispatch all requests to ourselves in order to be able to do the processing in separate threads'''
        return self


    def render(self, request):
        '''Dispatch the real work to a thread'''
        d = threads.deferToThread(self.do_work, request)
        d.addCallbacks(self.on_request_done, errback=self.on_error, callbackArgs=[request], errbackArgs=[request])
        #If the client aborts the request, we need to cancel it to avoid error messages from twisted
        request.notifyFinish().addErrback(self.on_request_failed, d)
        return NOT_DONE_YET


    def do_work(self, request):
        '''This method runs in thread context.'''
        db_session = self.DBSession()
        request.db_session = db_session
        user = db_session.query(User).first()

        body = 'Hello, {} '.format(user.login) * 1024 #generate some output data
        request.write(body)
        request.finish()


application = service.Application("My Testapp")
s = internet.TCPServer(8080, Site(WebInterface()), interface='0.0.0.0')
s.setServiceParent(application)

Answer 1

I've solved the issue. @beiller, you were pretty close to it with your guess. As can be seen in the source code of my question, the DB session gets opened after request processing has started, but the two are closed in the same (instead of the reverse) order. Close the session before calling request.finish(), and everything's fine.

Answer 2

Its possible you are not closing your database connection or some dead lock situation in the database using SQLAlchemy? I've had flask lock up on me before from not closing connections / not ending transactions.