CODEWITHSUNDEEP
pythonpython-3.xmultithreading
Denis Steinman
Denis Steinman

Reputation: 7799

How to wait any thread in ThreadPool?

My app gets new tasks infinitely, I have created a class that will handle all these incoming tasks:

class Executor:
    pool: ThreadPool

    def __init__(self, pool_size: int):
        self.pool = ThreadPool(pool_size)

    def start(self):
        while True:
            self.refresh_args()
            self.pool.map(self.handler, self.args)
            self.pool.join()

This code is wrong, of course. The problem is that I don't need to wait for all tasks in the pool. The Executor must add a new task to pool as soon as at least one thread will finish work. It will be endless loop and all threads in the pool must be busy always.

How to implement this logic? Or maybe should I look for another way without using ThreadPool? How is it implemented in other software?

Upvotes: 1

Views: 345

Answers (1)

JPery
JPery

Reputation: 69

You can do it by using a multiprocessing.Queue, passing the number of tasks as the max number of elements in the Queue.

When you put something in the queue, that thread keeps waiting until it is in the queue. At the same time, you can make your loop like

while True:
    queue.get() # blocks if queue is empty

and put every element in a new Thread:

class Executor:
    pool: ThreadPool

    def __init__(self, pool_size: int):
        self.elements = multiprocessing.Queue(pool_size)

    def start(self):
        while True:
            self.refresh_args()
            element = self.elements.get() # blocks if queue is empty
            # put element in new thread
            # when task is finished, put new element in queue

Upvotes: 2

Related Questions