CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessing
andrea m.
andrea m.

Reputation: 688

How many processes can be efficiently spawn concurrently on an 8-core machine?

I'm using python's multiprocessing module to open multiple concurrent processes on an 8-core iMac. Something puzzles me on the activity monitor. If I send 5 processes, each is reported using close to 100% CPU, but the CPU load is about 64% idle overall. If I send 8 concurrent processes, it gets to about 50% usage.

Am I interpreting the activity monitor data correctly, meaning that I could efficiently spawn more than 8 processes at once?

activity monitor screenshot

Upvotes: 0

Views: 1202

Answers (1)

hobbs
hobbs

Reputation: 240314

Presumably your "8-core" machine supports SMT (Hyperthreading), which means that it has 16 threads of execution or "logical CPUs". If you run 8 CPU-bound processes, you occupy 8 of those 16, which leads to a reported 50% usage / 50% idle.

That's a bit misleading, though, because SMT threads aren't independent cores; they share resources with their "SMT sibling" on the same core. Only using half of them doesn't mean that you're only getting half of the available compute power. Depending on the workload, occupying both SMT threads might get 150% of the throughput of using only one; it might actually be slower due to cache fighting, or anywhere in between.

So you'll need to test for yourself; the sweet spot for your application might be anywhere between 8 and 16 parallel processes. The only way to really know is to measure.

Upvotes: 4

Related Questions