CODEWITHSUNDEEP
pythonmultithreadingwebsockettwisted
Sunjay Varma
Sunjay Varma

Reputation: 5115

How to concurrently run several continuous tasks in a Python Twisted server using threading?

Let's say that you have 1000 clients connected to a twisted server. Those clients are constantly polling for updates (or are just persistently connected through something like a WebSocket). The server's job is to go through all 1000, check which of them have updates and then send those off as they come.

Is there anyway to make it so that, in Twisted, I could run 250 in 4 threads (or divide it up even further)? How does Twisted decide how many tasks to run at once? Can I check the 250 continuously and even change the distribution from time to time?

Here's a very simplistic version of what I imagine, though I have no idea how this looks in Twisted:

class Server:

    queues = [["250 clients"], ["250 clients"], ["250 clients"], ["250 clients"]]

    @classmethod
    def run_server(cls):
        for i in xrange(len(cls.queues)):
            # this would run in its own thread
            cls.continuously_check_for_updates(cls.queues[i])

        # server mainloop
        while True:
            if get_client_connection():
                store_client_in_queue()
                distribute_client_queues()

    @classmethod
    def continuously_check_for_updates(cls, queue):
        i = 0
        while True:
            if i >= len(queue):
                i = 0

            client = queue[i]
            if check_for_updates(client):
                report_update(client)

            i += 1

I appreciate your insight and advice. Thanks.

Upvotes: 1

Views: 407

Answers (1)

Jean-Paul Calderone
Jean-Paul Calderone

Reputation: 48325

No. Python will only execute a single thread at a time (ignoring some irrelevant details). There's no reason to use threading to solve this problem.

Instead, you should use an event-driven approach where you don't have to constantly examine a thousand different data structures. Instead, take action when the relevant change occurs.

For example, don't write a program like:

def addSomeData(data):
    someList.append(data)

def checkForData():
    if someList:
        return someList.pop(0)
    return None

while True:
    data = checkForData()
    if data is not None:
        processData(data)

Instead write a program like this:

def addSomeData(data):
    processData(data)

It's much simpler and more efficient.

Upvotes: 1

Related Questions