CODEWITHSUNDEEP
pythonmultithreadingtwistedpython-multiprocessing
user823743
user823743

Reputation: 2172

Twisted threads for multiple clients on server

I have implemented a server program using Twisted. I am using twisted.protocols.basic.LineReceiver along with twisted.internet.protocol.ServerFactory.

I would like to have each client that connects to the server run a set of functions in parallel (I'm thinking of multi-threading for this).

I have some confusion with using twisted.internet.threads.deferToThread for this problem.

  1. Should I call deferToThread in the ServerFactory for this purpose?
  2. Are twisted threads, thread-safe with respect to race conditions?
  3. Previously, I tried using multiprocessing in my server program but it seemed not to work in combination with the Twisted reactor, while deferToThread did the job.

I'm wondering how are Twisted threads implemented? Don't they utilize multiprocessing?

Upvotes: 1

Views: 711

Answers (1)

Jean-Paul Calderone
Jean-Paul Calderone

Reputation: 48335

  1. Previously, I tried using multiprocessing in my server program but it seemed not to work in combination with the Twisted reactor, while deferToThread did the job. I'm wondering how are Twisted threads implemented? Don't they utilize multiprocessing?

You didn't say whether you used the multi-threaded version of multiprocessing or the multi-process version of multiprocessing.

You can read about mixing Twisted and multiprocessing on Stack Overflow, though:

Mix Python Twisted with multiprocessing? Twisted network client with multiprocessing workers? is twisted incompatible with multiprocessing events and queues?

(And more)

To answer the shorter part of this question - no, Twisted does not use the stdlib multiprocessing package to implement its threading APIs. It uses the stdlib threading module.

  1. Are twisted threads, thread-safe with respect to race conditions?

The answer to this is implied by the above answer: no. "Twisted threads" aren't really a thing. Twisted's threading APIs are just a layer on top of the stdlib threading module (which is really just a Python API for POSIX threads (or something kind of similar but different on Windows). Twisted's threading APIs don't magically eliminate the possibility of race conditions (if there is any magic in Twisted, it is the ability to do certain things concurrently without using threads at all - which helps reduce the number of race conditions in your program, though it doesn't entirely eliminate the possibility of creating them).

  1. Should I call deferToThread in the ServerFactory for this purpose?

I'm not quite sure what the point of this question is. Are you wondering if a method on your ServerFactory subclass is the best place to put your calls to deferToThread? That probably depends on the details of your implementation approach. It probably doesn't make a huge difference overall, though. If you like the pattern of having the factory provide services to protocol instances - go for it.

Upvotes: 1

Related Questions