CODEWITHSUNDEEP
pythonmultithreadingopencv
Drew
Drew

Reputation: 229

Is there a way to re-use thread workers in python?

I'm writing some open cv code that could be easily parallelized. But I'm running into an issue. Examples of python multi-threading generally look like this:

from multiprocessing.dummy import Pool

def slow_function(input):
    ** opencv transforms and things **
    return output

worker_pool = Pool(4)
result = worker_pool.map(slow_function,list_of_inputs)
worker_pool.close()
worker_pool.join()

I've gotten this to work for my code, but I've been timing it and the line worker_pool = Pool(4) takes 100ms alone. The rest of the code runs very fast.

This creates a minimum overhead of 100ms, which makes in not particularly useful. My application is real time, so I would be capped at 10fps even if the rest of my code ran instantly.

It seems like I should be able to avoid creating and destroying the pool every cycle, thus saving those 100ms. Is there a way to do this? Ideally, I would like to call map then join, and just have the workers "forget" that they even ran anything once they run slow_funcion and return

Upvotes: 0

Views: 67

Answers (2)

JohanL
JohanL

Reputation: 6891

The simplest solution to your problem is to not call worker_pool.close() and/or worker_pool.join() at all. That will keep your threads alive and available to process new data.

worker_pool.map() will take care of waiting for the threads to deliver a result anyway and will give you a synchronization point in your code.

You do need to keep your thread pool alive and available, though. Depending on your architecture, that may or may not be best made by wrapping it in an object (thus defining a class).

Upvotes: 1

Nulljack
Nulljack

Reputation: 21

You could just spawn processes as need and use the Queue module to pass data between your main process and the "slow_function" process.

Upvotes: 0

Related Questions