Reputation: 19379
There are 101 things stored in THINGS
variable.
The code declares 101 threads and executes them all at once instantly all at the same time.
I wonder if we can limit the number of the active threads to just 12.
At first only 12 threads should pick their 12 THINGS to process. The rest of the threads should be waiting for the first 12 to finish their jobs. When the first 12 threads are all done then the next 12 threads would pick up the next 12 THINGS to process. And so one.
Would it be possible?
import Queue
import threading, time
class MyThread(threading.Thread):
def __init__(self, theQueue=None):
threading.Thread.__init__(self)
self.theQueue=theQueue
def run(self):
thing=self.theQueue.get()
self.process(thing)
self.theQueue.task_done()
def process(self, thing):
time.sleep(1)
print 'processing %s'%thing.name
queue=Queue.Queue()
THINGS = ['Thing%02d'%i for i in range(101)]
THREADS=[]
for thing in THINGS:
thread=MyThread(theQueue=queue)
thread.name = thing
THREADS.append(thread)
thread.start()
for thread in THREADS:
queue.put(thread)
Upvotes: 2
Views: 4723
Reputation: 19379
The working solution is posted below. The basic idea is that we declare only as many Threads instances as there are available CPUs. Then we proceed by adding the "tasks" (or "things" here) to the Queue. As soon as the task is added to the queue it is being immediately picked up by one of the Thread instances we declared in the previous step.
Important: In order for this mechanism to work the MyThread.run()
method should be running inside of the while
loop. Otherwise MyThread instance will be terminated as soon as it completes the very first task. The while
loop will exit itself after no tasks in the Queue are left. That is the end of story.
import Queue
import threading, time
class MyThread(threading.Thread):
def __init__(self, theQueue=None):
threading.Thread.__init__(self)
self.theQueue=theQueue
def run(self):
while True:
thing=self.theQueue.get()
self.process(thing)
self.theQueue.task_done()
def process(self, thing):
time.sleep(1)
print 'processing %s'%thing
queue=Queue.Queue()
THINGS = ['Thing%02d'%i for i in range(101)]
AVAILABLE_CPUS=3
for OneOf in range(AVAILABLE_CPUS):
thread=MyThread(theQueue=queue)
thread.start() # thread started. But since there are no tasks in Queue yet it is just waiting.
for thing in THINGS:
queue.put(thing) # as soon as task in added here one of available Threads picks it up
Upvotes: 4