Making Asynchronous Calls Using Twisted Inside a Django Project

Question

I have a Django web server that accepts requests from clients. In addition, I have another server, written with Twisted. I wrote a Twisted protocol - its server "half" is running on the server, and I have a very easy to use client protocol as well.

Occasionally the Django server receives requests that need to initiate a request to the Twisted server. This may obviously be a long operation, so it cannot be synchronous. I'm having trouble incorporating a Twisted client inside the Django project to make these requests.

Simplifying the architecture a bit, it looks something like this:

Initially, I created a global thread instance in the views.py file of my Django project that runs Twisted's reactor loop, and whenever a request was received through the Django server, I'd add it to the reactor asynchronously to be performed by that thread. This worked fine with the python manage.py runserver script, however when I ran it on my Apache server I would sometimes get the requests on the Twisted server and sometimes wouldn't.

I'm assuming this is because Apache also runs several threads, each of which tries to create that reactor-thread. Since there may only be one reactor per process, only one thread succeeds while the others fail, therefore only some requests make it through - the ones in the "winning" thread (this is just a theory, I haven't verified it).

I believe my options are:

1. Somehow share my reactor-thread instance among all Apache threads so that there's really just one reactor. Not sure exactly how to do this - everything I find online talks about sharing data (through Django's DB), not sharing instances.
2. Run the Twisted client as a separate process on the Django server machine, and send requests from Django with some RPC method (that is not Twisted-based - otherwise I'd be back in square one of running a Twisted client inside Django, which I cannot seem to be able to do...).

Any advice on what is the correct solution / how to implement it? Is there another way I'm not thinking of?

Thank you.

Answer 1

Use Crochet.

Stack Overflow thinks this answer isn't sufficient to solve your problem, because it isn't enough words. But check it out; Crochet does exactly what you need.