CODEWITHSUNDEEP
pythonmultithreadingasynchronoustwisted
user974896
user974896

Reputation: 1813

Threads vs Asynchronous Networking (Twisted) Python

I am writing an implementation of a NAT. My algorithm is as follows:

  1. Packet comes in
  2. Check against lookup table if external, add to lookup table if internal
  3. Swap the source address and send the packet on its way

I have been reading about Twisted. I was curious if Twisted takes advantage of multicore CPUs? Assume the system has thousands of users and one packet comes right after the other. With twisted can the lookup table operations be taking place at the same time on each core. I hear with threads the GIL will not allow this anyway. Perhaps I could benifit from multiprocessing>

Nginix is asynchronous and happily serves thousands of users at the same time.

Upvotes: 4

Views: 2111

Answers (1)

andrean
andrean

Reputation: 6796

Using threads with twisted is discouraged. It has very good performance when used asynchronously, but the code you write for the request handlers must not block. So if your handler is a pretty big piece of code, break it up into smaller parts and utilize twisted's famous Deferreds to attach the other parts via callbacks. It certainly requires a somewhat different thinking than most programmers are used to, but it has benefits. If the code has blocking parts, like database operations, or accessing other resources via network to get some result, try finding asynchronous libraries for those tasks too, so you can use Deferreds in those cases also. If you can't use asynchronous libraries you may finally use the deferToThread function, which will run the function you want to call in a different thread and return a Deferred for it, and fire your callback when finished, but it's better to use that as a last resort, if nothing else can be done.

Here is the official tutorial for Deferreds:

http://twistedmatrix.com/documents/10.1.0/core/howto/deferredindepth.html

And another nice guide, which can help to get used to think in "async mode":

http://ezyang.com/twisted/defer2.html

Upvotes: 5

Related Questions