CODEWITHSUNDEEP
pythonmultithreadingcpu-usageprocessor
kkk
kkk

Reputation: 1920

How should be spawn threads in python

I am running a python program in a server having python2.7.6 . I have used threading module of python to create multiple threads . I have created 23 threads , so i am confused whether all my processor cores are actually being used or not , Is there any way i can check this in my server . Any suggestion as what should be the ideal number of threads that should be spawned according to the number of processors that we have in order to improve efficiency of my program.

Upvotes: 1

Views: 748

Answers (2)

nlloyd
nlloyd

Reputation: 2036

David Beazly has a great talk on threading in Python. He also has a great presentation on the subject here.

Unfortunately, Python has something called the GIL which limits Python to executing a single thread at a time. To use all your cores you'd have to use multiple processes: See Multiprocessing.

Some in the Python community don't necessarily look at the GIL as a setback, you can utilize multiple cores through other means than shared-memory threading.

Look here for a great blog post on utilizing multiple cores in Python.

Edit: The above is true for CPython (the most common and also the reference implementation of Python). There's a few "yes but" answers out there (mostly referring to multithreading on a different Python implementation) but I'd generally point people to answers like this one that describe the GIL in CPython and alternatives for utilizing multiple cores6

Upvotes: 1

ffff
ffff

Reputation: 3070

There is no real answer to this question and everyone might have a different view. Determining the number of threads for your application to improve performance should be decided after testing for x scenarios with y of threads. Performance depends on how the OS scheduler is scheduling your threads to the available CPU cores depending on the CPU load or number of processes running. If you have 4 cores, then it doesn't mean it's good to execute 4 threads only. In fact you can run 23 threads too. This is what we call the illusion of parallelism given to us by the scheduler by scheduling processes after processes hence making us think everything is running simultaneously.

Here is the thing

If you run 1 thread, you may not gain enough performance. As you keep increasing threads to infinity, then the scheduler will take more time to schedule threads and hamper your overall application performance.

Upvotes: 0

Related Questions