CODEWITHSUNDEEP
pythonpython-3.xproc
nz_21
nz_21

Reputation: 7393

How to get number of workers available in ProcessPoolExecutor?

I am running some cpu bound code inside an async function, using ProcessPoolExecutor

pool = concurrent.futures.ProcessPoolExecutor(10)
...
loop.run_in_executor(
            pool, cpu_bound_func)

How do I get the number of workers available to be used?

Upvotes: 1

Views: 1702

Answers (1)

Sébastien Lavoie
Sébastien Lavoie

Reputation: 907

The Python docs for ProcessPoolExecutor say that "If max_workers is None or not given, it will default to the number of processors on the machine."

If we peek under the hood, we can see that ProcessPoolExecutor uses os.cpu_count() to determine the number of workers to use (self._max_workers = os.cpu_count() or 1). So one way of getting the number of workers is to do the following (not recommended since the implementation of _max_workers could change in the future):

pool = ProcessPoolExecutor()
print(pool._max_workers)

If you need this information before creating the pool, you can use os.cpu_count() directly:

import os
from concurrent.futures import ProcessPoolExecutor

actual_cpu_count = os.cpu_count()
desired_cpu_count = actual_cpu_count - 1  # for example
pool = ProcessPoolExecutor(desired_cpu_count)

Upvotes: 0

Related Questions