CODEWITHSUNDEEP
pythonmultithreadingpython-multithreading
delica
delica

Reputation: 1707

Limiting number of concurrent threads

What is the easiest way to make sure this code runs only 8 threads at a time. I need it to keep running and reusing threads. If one thread finishes, it should start another in its place immediately.

threads = []
for user in user_list:
    thread = threading.Thread(target=parse_func, args= self,user,thread_name,), name= thread_name)
    thread.start()
    threads.append(thread)
for t in threads:
    t.join()

Upvotes: 0

Views: 797

Answers (3)

delica
delica

Reputation: 1707

I ended up spawning the exact number of threads I needed, let's say a 100:

for i in range(100):
    Thread(target=get_url).start()

Since I wanted the threads to each keep alive and keep checking the processing queue, I used an infinite loop inside each thread, so that the 100 threads keep running. I could not use a thread pool since it would shutdown as soon as the queue was exhausted the first time - if I refilled the queue with more work an hour later, the thread pool would have already shut down at that point so I had to make sure to create a new pool etc. At this point it becomes easier to maintain my own pool. 

def get_url():
    while True:
        item = q_worker.get()
        #do work with item
        q_worker.task_done()

Upvotes: 1

Solomon Slow
Solomon Slow

Reputation: 27115

You want a thread pool.

The idea of a thread pool, is that instead of creating new threads, your application code creates new tasks, and then it submits the tasks to the thread pool. The pool consists of some number of threads (maybe a variable number, maybe a fixed number, depends how its implemented) and a blocking queue.

The client program puts tasks onto the blocking queue, while each of the pool threads sits in a loop, taking tasks from the queue and performing them.

A simplistic thread pool might have a fixed set of threads, and it might run forever. A sophisticated thread pool might have means to spin up or shut down threads in response to changing demand and/or changing system load.

I don't have enough Python experience to know whether there is a standard thread pool interface that everybody uses or, to recommend any existing thread pool implementation.

You can always write your own. A thread pool with eight fixed threads that run forever would not be hard to make.

Upvotes: 1

user1252280
user1252280

Reputation: 371

You may want to look into semaphores and use it like this:

threading.BoundedSemaphore(maximumNumberOfThreads)

Taken from the documentation, semaphores are usually used to guard resources with limited capacity, for example, a database server.

Another example is the following, taken from the documentation: Before spawning any worker threads, your main thread would initialize the semaphore:

maxconnections = 5
...
pool_sema = BoundedSemaphore(value=maxconnections)

Once spawned, worker threads call the semaphore's acquire and release methods when they need to connect to the server:

pool_sema.acquire()
conn = connectdb()
... use connection ...
conn.close()
pool_sema.release()

Upvotes: 1

Related Questions