CODEWITHSUNDEEP
pythonpython-3.xconcurrent.futures
edjohnsonwilliams
edjohnsonwilliams

Reputation: 480

Number of max_workers when using ThreadPoolExecutor from concurrent.futures?

What are the factors to consider when deciding what to set max_workers to in ThreadPoolExecutor from concurrent.futures?

As long as you can expect Python 3.5+ to be available, is there any reason not to set max_workers to None which will then "default to the number of processors on the machine, multiplied by 5" as described in the docs here? https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor

Upvotes: 38

Views: 29709

Answers (1)

J0ANMM
J0ANMM

Reputation: 8531

I don't think this question can be so generically solved; it will depend on each case.

From this answer:

The more threads you use, the higher concurrency you'll achieve (up to a point), but the less CPU cycles you'll get (as there will be context switches). You have to instrument your application under typical workloads to see what works best for you. There is no universally optimal solution for this.

Upvotes: 24

Related Questions