Twisted Python - In deferred's callback's function call, raised exception not displayed in stdout

Question

Test Setup

I have an asynchronous Twisted HTTP server for handling POST:

class HttpResource(Resource):
    isLeaf = True;
    def render_POST(self, request):
        ...

if __name__ == "__main__":
    factory = Site(HttpResource())
    reactor.listenTCP(8000, factory)
    reactor.run()

I created a simple test() function that raises an exception:

def test():
    print "test() called"
    raise Exception("Exception raised!")

Capturing Exception message (i.e. "Exception raised!") in program output

If I call test() within render_POST(), the exception is raised and output shows the message as expected:

Traceback (most recent call last):
    raise Exception("Exception!")
exceptions.Exception: Exception!

However, if within render_POST() I create a deferred pointing to a callback that executes test(), the exception is presumably still raised (client gets a 500 internal server error) but no exception error message (the "Exception raised!" phrase) is shown.

Question

How can I get Exception error messages to show when they arise from function calls within a callback?

UPDATE 1: Demo code provided (requires Twisted for server.py)
**UPDATE 2: Demo code returns "NOT_DONE_YET"

client.py

import json, urllib2

if __name__ == "__main__":
    while True:
        cmd = raw_input("Enter cmd number (1, 2, or 0): ")
        if cmd == str(1):
            raw_data = {'cmd' : 'cmd1'}
        elif cmd == str(2):
            raw_data = {'cmd' : 'cmd2'}
        elif cmd == str(0):
            break;
        req = urllib2.Request('http://localhost:8000')
        req.add_header('Content-Type', 'application/json')
        response = urllib2.urlopen(req, json.dumps(raw_data))
        print response.read()        

server.py

import json, time
from twisted.internet import reactor
from twisted.web.client import getPage
from twisted.web.resource import Resource
from twisted.web.server import Site, NOT_DONE_YET

def test():
    print "test() called"
    time.sleep(1)
    raise Exception("Exception raised!")

def callback(result):
    print "callback() called"
    test()

class HttpResource(Resource):
    isLeaf = True;
    def render_POST(self, request):
        msg = json.loads(request.content.getvalue())
        if msg['cmd'] == 'cmd1': # call test() directly
            test()
        elif msg['cmd'] == 'cmd2': # call test() from callback
            d = getPage('http://www.yahoo.com')
            d.addCallbacks(callback, callback)

        return NOT_DONE_YET

if __name__ == "__main__":
    factory = Site(HttpResource())
    reactor.listenTCP(8000, factory)
    reactor.run()

Answer 1

There are two problems here relevant to your missing traceback.

The first problem with your program is that when you return None from render_POST, this tells twisted.web that you are done handling the request and that the connection should be closed immediately. By the time your callback gets around to pushing some data out to the HTTP channel, the response has already been sent and the data is discarded.

The second problem is that you are never actually pushing data out to the connection at all. When render_POST itself raises an exception, the code calling it can catch that exception. However, the code calling render_POST doesn't expect a Deferred, and even if it did, you're not returning one here.

There is also a third problem, which is that you're using time.sleep which freezes your whole program rather than deferLater which would return a Deferred that simply fires at a later time and allows concurrent tasks to proceed. Since the time.sleep is entirely beside the point though, and the behavior is the same whether you include it or not, we can ignore it :-).

There is a way to address this problem within Twisted itself, but an easier way to address it would be to use Klein, which handles Deferred for you automatically.

Using Klein, your example would be:

import json

from klein import run, route
from twisted.web.client import getPage

def test():
    print "test() called"
    raise Exception("Exception raised!")

def callback(result):
    print "callback() called"
    test()

@route("/", methods=["POST"])
def example(request):
    msg = json.loads(request.content.getvalue())
    if msg['cmd'] == 'cmd1': # call test() directly
        test()
    elif msg['cmd'] == 'cmd2': # call test() from callback
        d = getPage('http://www.yahoo.com')
        return d.addCallbacks(callback, callback)

run("localhost", 8000)

Note the return d.addCallbacks; if you don't return the result it will continue failing in more or less the same way it does now.