CODEWITHSUNDEEP
pythonmultiprocessing
Mark Smith
Mark Smith

Reputation: 767

Python Multiprocessing - coding for a dual, quad and six core cpus

I have a quick question on Python multiprocessing.

I am writing some code which will use multiprocessing to split tasks between cores/virtual cores but I have a number of machines and each has a different CPU...

I have an Intel Centrino2 dual core, an Intel i5 quad core and a dual Xeon hex core (with hyper threading) machine that will run this code.

This means that each machine will either have 2, 4 or 12/24 cores and I want to know if the same code will run on all machines or if I will need to tailor it for the specific number of cores on each machine...

I am thinking that the (upto) 10 python processes my code will create will run on a physical/virtual core if one is available but if the machine only has 2 or 4 cores, then the processes will run 'time-sliced' on the available cores similar to as they would if I were using threading - is this correct?

I will must likely 'suck it and see' but I am interested in any advice or information on this topic that you can suggest...

I am planning to run the code on 64 bit Centos Linux and Windows 10 OSes and use Anaconda Python 3.6 in case this is useful.

Thanks

Upvotes: 1

Views: 1609

Answers (1)

Mark Smith
Mark Smith

Reputation: 767

I found this Q/A on Stackoverflow which supports what I had originally thought - the code will run on any of the 3 cpus so will not need to be tailored on a per cpu basis...

python multiprocessing and number of cores

Windows vs Linux performance:

As a side note to the above, there are some performance issues around spawning python multiprocessing processes on Window vs Linux, as in Windows, processes are 'heavy' and take more time/resources to create than on Linux platforms - this may mean that a python program using multiprocessing performs slightly better on Linux vs Windows.

Sharing data between processes via a multi-process queue (Overview):

Python multiprocessing differs from threading in as much as each process runs in its own address space so any data contained within a process isn't available to other processes by default - here is a good video which explains how to share data between processes using queuing.

https://www.youtube.com/watch?v=sp7EhjLkFY4

Upvotes: 1

Related Questions