How can I wait for any one thread out of several to finish?

Question

Say I want to launch multiple threads to perform tasks, and as they finish I want to process their work units. And optionally launch more threads to do more work units.

Is there any standard way to tell Python that I want to wait for any one of my threads to complete? Bonus points if it's easy to know which one.

Googling the question produced no useful results. All the answers were how to wait for all the threads, or how to wait for the first thread. Neither of these are what I'm looking for.

I've toyed with using a counting semaphore, but is there a better way?

Answer 1

Adding on: I tried using a semaphore, and it worked fine, but you don't know which thread terminated.

A Queue would also work. Each thread puts its result onto the queue when finished, and the caller removes items as they become available.

In both cases, caller is responsible for knowing how many threads to wait for.

Leaving the question open for now, in case someone comes up with a better answer.

Answer 2

haha, use to have the same problem.

try to use the concurrent.futures.ThreadPoolExecutor along with as_completed to deal with it.

Use ThreadPoolExecutor to instantiate a thread pool object. Pass the max_workers parameter to set the maximum number of threads that can run concurrently in the thread pool.

you can use such as submit,cancel,result to check if a task is finished, they require constantly polling in the main thread, which can be inefficient. Sometimes, we only need to know when a task finishes in order to fetch its result, rather than continuously checking each task. This is where the as_completed() method comes in handy, as it allows retrieving the results of all tasks once they are completed. The as_completed() method is a generator that blocks until a task is completed. When a task finishes, it yields the task, allowing the main thread to process it. After processing, the generator will continue blocking until all tasks are finished. As shown in the results, tasks that complete first will notify the main thread first.