CODEWITHSUNDEEP
python
Vladimir Protasov
Vladimir Protasov

Reputation: 302

Threads in Python again

guys!
My application is a bot. It simply receives a message, process it and returns result.
But there are a lot of messages and I'm creating separate thread for processing each, but it makes an application slower (not a bit).
So, Is it any way to reduce CPU usage by replacing threads with something else?

Upvotes: 0

Views: 272

Answers (4)

jpsimons
jpsimons

Reputation: 28090

You might not even need threads. If your server can handle each request quickly, you can just make it all single-threaded using something like Twisted.

Upvotes: 0

Thomas Ahle
Thomas Ahle

Reputation: 31604

Threads and processes have the same speed. Your problem is not which one you use, but how many you use.

The answer is to only have a fixed couple of threads or processes. Say 10. You then create a Queue (use the Queue module) to store all messages from your robot. The 10 threads will constantly be working, and everytime they finish, they wait for a new message in the Queue.

This saves you from the overhead of creating and destroying threads. See http://docs.python.org/library/queue.html for more info.

def worker():
    while True:
        item = q.get()
        do_work(item)
        q.task_done()

q = Queue()
for i in range(num_worker_threads):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done

Upvotes: 1

user97370
user97370

Reputation:

You probably want processes rather than threads. Spawn processes at startup, and use Pipes to talk to them.

http://docs.python.org/dev/library/multiprocessing.html

Upvotes: 3

Marek Sapota
Marek Sapota

Reputation: 20591

You could try creating only a limited amount of workers and distribute work between them. Python's multiprocessing.Pool would be the thing to use.

Upvotes: 0

Related Questions