CODEWITHSUNDEEP
pythonmultiprocessingpool
Andrew Smith
Andrew Smith

Reputation: 197

Proper way to execute ProcessingPools in sequence

I am using the pathos.multiprocessing library (since my worker methods have lambdas in them), and am trying to run two ProcessingPools in sequence.

For example:

from pathos.multiprocessing import ProcessingPool
pool=ProcessingPool(nodes=2)
res=pool.map(workerFunc,workerArgList)
pool.close()
pool.join()
# ...
pool=ProcessingPool(nodes=2)
res=pool.map(workerFunc2,workerArgList2)
pool.close()
pool.join()

The code, as is, returns an AssertionError in pool.py's map() function at the 2nd pool.map() (at assert self._state == RUN). Each pool works fine independently. It appears that, for some reason, the pool's state is set to TERMINATE instead of RUN in the 2nd initialization. How should I initialize the 2nd pool (or terminate the first) to get the desired behavior? Or is this some sort of bug I have stumbled upon?

Thank you! Andrew

Upvotes: 1

Views: 192

Answers (1)

Vito
Vito

Reputation: 1698

According to the official Python Documentation on Pool.map(),

It blocks until the result is ready.

In other words, pool.map() won't return the result until all processing is finished. When using pool.map() you don't need to run pool.close() and pool.join() every time, just do it once after you're done using pool.

Upvotes: 1

Related Questions