CODEWITHSUNDEEP
parallel-processingmpiopenmpi
bob.sacamento
bob.sacamento

Reputation: 6651

In OpenMPI, is there a way to put jobs on specific cores?

The question pretty much says it all. This is for benchmarking purposes. I really do need to target specific cores on specific nodes. Targeting particular nodes is not enough in and of itself for this project. I am told that you can do this with a hostfile, but the only documentation I can find on hostfiles discusses only telling OpenMPI how many slots are available on a node, not which specific slot(s) to use.

Upvotes: 0

Views: 48

Answers (2)

Hristo Iliev
Hristo Iliev

Reputation: 74465

Open MPI's launcher mpirun / mpiexec has the --cpu-list option which allows you to specify a list of processor IDs to bind to. If you simply need to tell the library which slots on a given host to give to which rank, you need to use a rankfile instead of a hostfile. Rankfiles are like inverted hostfiles that tell Open MPI where to place each rank. You can find more in the official documentation, but the gist of it is:

  1. Create a rankfile listing all ranks and their hosts and the slots you'd like each rank to run on:

    rank 0=host1 slot=0-3
rank 1=host1 slot=4,5
rank 2=host2 slot=0,1,2
...

    Slots are specified as comma-separated lists where each element is either a single number or a range. You can also prefix with a package ID. The numbers are the logical CPUs as understood by hwloc, the system topology library of Open MPI. Instead of fixed host names, you can use offsets in the host list with +nX, e.g., +n0 instead of host1 in the example above.

  2. Call mpirun with --map-by rankfile:file=/path/to/rankfile. If you'd like to target specific hardware threads, you should also append :hwtcpus to tell hwloc that you'd like the numbering to also reflect the different hardware threads on each core.

Upvotes: 1

jcernuda95
jcernuda95

Reputation: 7

Sure, you are looking for what is called thread affinity control.

Volume 2 of the art of HPC, chapter 25 (online for free) has a great explanation on how to do it, here is a global view answer:

Thread placement can be controlled with two environment variables:

The environment variable OMP_PROC_BIND describes how threads are bound to OpenMP places

While the variable OMP_PLACES describes these places in terms of the available hardware.

When you're experimenting with these variables it is a good idea to set OMP_DISPLAY_ENV to true, so that OpenMP will print out at runtime how it has interpreted your specification.

Upvotes: -2

Related Questions