CODEWITHSUNDEEP
pythonmultiprocessingpool
Alberto Rolandi
Alberto Rolandi

Reputation: 13

How to tell Pool to use a certain number of cores per process?

I'm using multiprocessing.Pool to parallelise some computation in a project. How can I tell Pool to use n (eg. 4) cores per parallel process?

Say I have 8 cores. Will this code make sure that each parallel process is running on 4 cores?

from multiprocessing import Pool

def fun(in):
    print(in)

pool = Pool(2)
pool.map(fun, [1, 2, 3, 4, 5, 6])

Upvotes: 0

Views: 2313

Answers (2)

user10186512
user10186512

Reputation: 453

multiprocessing.Pool will not create multiple threads per process, but multiple single-threaded processes. "Parallel processes" means multiple processes that run in parallel, not single processes that are somehow internally parallel.

Each process in a multiprocessing.Pool will run on exactly one core at a time, so you should create as many processes as you want cores to be utilised - in this case if you want to potentially utilise all eight cores, you want eight processes in the pool:

pool = Pool(8)

You can also not pass an argument at all, and Pool will automatically allocate as many processes as you have CPU cores.

Documentation for multiprocessing.Pool:

processes is the number of worker processes to use. If processes is None then the number returned by os.cpu_count() is used.

Note, however, that you cannot actually tell Pool to use a specific core or specific amount of cores - that decision is made by your operating system, which will usually try to allocate workloads evenly across cores.

Upvotes: 1

quamrana
quamrana

Reputation: 39414

No, your code will allow Pool to create two processes (they use one core each) and map() will process your collection of items through the designated function in two streams.

I think you might mean:

pool = Pool(4)

This will mean that your fun will be running on 4 cores simultaneously.

Upvotes: 0

Related Questions