CODEWITHSUNDEEP
pythonmultithreadingpython-multithreading
Tomasz Kapłoński
Tomasz Kapłoński

Reputation: 1378

How can I determine sensible thread number in python?

I am writing a simple script which should do big amount of checks. Every check is independent so I decided to put it into multiple threads. However I don't know how fast will be machine on which the script will be set. I've already found quite nice util to check basic parameters of the target machine but I am wondering if there's any way to determine what is the max sensible amount of threads (I mean the moment when new thread starts slowing the process instead of speeding it)?

Upvotes: 3

Views: 3406

Answers (3)

Pete
Pete

Reputation: 68

Threads for the purpose of speed in python is not a terribly good idea, particularly for cpu bound operations. The GIL sees off any potential performance improvements from multiple CPU's (the # of which is the theoretical limit of your speed increase from threading - though in practice YMMV).

For truly independent "checks" you are far better off looking at multiprocessing.

Upvotes: 2

danodonovan
danodonovan

Reputation: 20373

You can find out the number of cores your target machine has with 

import multiprocessing

multiprocessing.cpu_count()

If you choose multiprocessing to manage your tasks then you can set the Pool size, or number of worker threads dependent on system load and .cpu_count().

As to what is a good number for you to choose for your program, you will have to decide for yourself :-)

Upvotes: 4

Cris Stringfellow
Cris Stringfellow

Reputation: 3808

You might have your answer in your last sentence. You could measure the execution time, as it changes with thread additions. And adaptively add or remove (and requeue the removed thread's job) threads to preserve an execution time you like. If you want to get sophisticated check out control theory.

Upvotes: 0

Related Questions