CODEWITHSUNDEEP
pythonmultiprocessingpool
Mario Kadastik
Mario Kadastik

Reputation: 33

Dynamic worker pool management in python multiprocessing

What I'd like to do is monitor how the system resource usage is and dynamically increase/decrease the amount of workers in the pool.

I namely have a 24 core node with 48GB of RAM and what I do is read in a 4GB datafile that I need to filter through. The system is also used by others so the available memory varies with time. As the 4GB input data is replicated to all workers (I've yet to find a way how to avoid that as I only need it in read-only in the workers, recommendations welcome, it's a dict() of dicts() and lists) that means I cannot spawn 24 workers as I'd immediately run out of memory. So what I'd like to do is start the process with say a safe number of 6 workers and then observe memory usage and spawn additional workers to the pool and if the memory usage gets high reduce the number of workers (i.e. allow some to complete, but not spawn new tasks into those workers anymore). This way I could maximally utilize the node while keeping RAM usage <95% say.

The reason this would be efficient is because the whole code runs for a few hours even with 6-12 parallel workers so if I can even for some time increase the number of workers 20-30% it'd reduce the total execution time.

Right now I've used both Pool.map() and Pool.apply_async() methods to send tasks to the pool so I have no direct preference if one method works better than the other.

Thanks in advance for recommendations.

Upvotes: 3

Views: 3226

Answers (1)

unutbu
unutbu

Reputation: 880707

The multiprocessing module provides a way for multiple processes to share a dict: 

manager = mp.Manager()
d = manager.dict()

Perhaps by using a shared dict, you could spawn 24 workers and still use far less memory. When one worker accesses or mutates the dict, other workers will block if they are also attempting to access or mutate the dict, but if this is not the primary bottleneck, being able to use 24 workers could dramatically reduce your execution time.

Upvotes: 3

Related Questions