CODEWITHSUNDEEP
multithreadingpython-3.xmultiprocessing
Mark Smith
Mark Smith

Reputation: 767

Python 3.6: Will running the same multi-threaded python script in separate CMD sessions get around Pythons GIL issue

I have a Python 3.6 script which is multi-threaded and runs on a windows 10 machine with 12 cores.

I want to run multiple instances of the script but I am worried about Pythons GIL issue in terms of the performance of each instance of the script.

Would doing any of the below work around this as I am under the impression that when I run an instance of the script, the Python process it runs within is running on just one CPU core and each thread called from within the script 'time slices' on that one core...

Therefore would:

A: Starting each instance of the script in its own CMD window allow the OS to automatically deal with starting each scripts parent Python processes on its own core and prevent any locking taking place...

or

B: Starting each CMD session via a shortcut that sets its affinity to a specific core and then run the Python script so that the python process and its threads run on the specific core the CMD process has been set to use...

or

C: My understanding of how threading and the Python GIL works is not correct and I need understand...

Any help will be much appreciated.

Upvotes: 0

Views: 148

Answers (1)

Solomon Slow
Solomon Slow

Reputation: 27125

I want to run multiple instances of the script

Then go ahead and do it!

I am worried about Pythons GIL issue in terms of the performance of each instance of the script.

The "GIL issue" is only an issue within any single multi-threaded Python process: No more than one thread belonging to the same Python process will ever execute Python code at the same instant. But, each Python process has its own GIL, and nothing prevents a thread in one Python process from running at the same time as a thread in some other Python process.

would (A) Starting each instance of the script in its own CMD window [...run each] Python processes on its own core.

No, because it is not normal for any process on an SMP platform (i.e., most multi-processor systems) to have "its own core." Cores are a resource that the operating system uses to run threads/processes, and applications normally have no reason to know or care which core is running which thread/process at any given moment in time.

(B) Starting each CMD session via a shortcut that sets its affinity to a specific core.

Some operating systems let you do that, but usually there is no benefit unless your application has very special needs. Normally, you should just trust the operating system scheduler to schedule processes and threads as efficiently as possible.

Upvotes: 3

Related Questions