mpi and process scheduling

Question

Let's say that the number of processes I'm launching are greater than the number of cores I'm working with. When a series of processes on a set of cores complete, I want to utilize those cores. Is there any way for me to do that?

I thought of updating my rankfile on the go, but I'm not sure if that will work.

Any input will be appreciated. Thanks!

Answer 1

Launching more MPI processes than the number of CPU cores available is often referred to as oversubscription. This is normally perfectly well supported by the MPI libraries and operating systems, but might require some tweaking at job's submission time. The main point one should be careful with is the process-to-core attachment possibly performed by the MPI job launcher (ie mpirun, mpiexec, ortrun, srun, prun, mpprun, [addYourPreferredLauncherHere], ...).

If process-to-core attachment is enabled, then the oversubscription is likely to be quite ineffective (understanding that it is already likely to be counter-effective to oversubscribe, even in the best possible running conditions). So you will have to simply refer to the documentation of your MPI launcher to see how to disable attachment (sometimes referred to as "process affinity") and just run your MPI code as usual, with simply more processes than there are cores. No modification in the MPI code itself is required.