How to optimize number of threads per number of cores

Question

I'm trying to get a better idea of how many threads should run on n number of cores. I know this a complicated question in which the answer depends on a number of factors, such as how much shared state there is, and how much sleeping and waiting on resources each thread does.

To simplify things, let's say we have 2 cores and just one process that can divide its work into threads with no shared state. Let's say each thread just performs computation after computation with no sleeping and no waiting on resources. Would the ideal number of threads in this case be 2?

Let's complicate things a bit and say that the threads have to do some sort of disk I/O. How does this change our answer? I would think that we could have more than 2 cores in this case.

Or let's say they don't do any sleeping or waiting on resources, but instead there's some memory that they both have access to that requires synchronization. How does this change our answer? I would think that in this case we may actually prefer 1 thread over 2, depending on how much synchronization is required.

Answer 1

When it comes to IO you don't think in terms of threads per core. You think in terms of threads per physical device. Each individual device has different optimal DOP's (magnetic disk = 1, SSD at least 4, network much higher).

For CPU bound work the optimal number is 1 (per core).

In mixed cases or cases where it is more complicated than that no general answer can be given. The system can behave in surprising ways (like collapsing under load!). The approach here is to test different DOPs and use the best. Generally, there will be exactly one optimum while both 1 and infinity perform much worse. So you only need to find the single maximum which is quite easy.

Answer 2

This is a hard question to answer in the general matter. It really depends on more specifics in the case. What to remember is that it costs to do context-switches - if you're only doing computations it would be wasteful to have 2 threads running on one core (as you wouldn't really gain anything - only lose in the context switches). On the other hand if you are waiting for resources and at the same time can continue with other calculations, it is a good idea to have a thread wait for those resources to not make the entire execution lag behind.