CODEWITHSUNDEEP
parallel-processingmpihpc
John Snow
John Snow

Reputation: 85

Group mpi process based on a flag

I need to create a sub-communicator (mi_comm_world_2) based on a bigger mpi communicator (mpi_comm_world).

In particular, after a detection loop such as

if something exists inside the proc proc

I need to collect in the new communicator mpi_comm_world_2 all the process flagged as true in respect of the check.

I am not able to find a clear documentation in order to do this job.

Upvotes: 3

Views: 765

Answers (1)

Richard
Richard

Reputation: 61289

Good question!

This is what the MPI_Comm_split command is good for.

Definition

int MPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm *newcomm)

Arguments

  • comm: Handle to the communicator you want to want build the new communicator from

  • color: A nonnegative integer indicating how to group the processes in the new communicators. Processes with the same color are in the same new communicator

  • key: An integer that controls rank assignment. Ranks are always assigned from 0 to the number of processes in the communicator. The key determines the relative ordering of the processes' ranks in the new communicator.

  • newcomm: The new communicator

  • int: The function returns an integer indicating whether it was successful or not.

Longer Explanations

Throughout this bear in mind that you have many processes executing what appears to be the same code. As such, the value of newcomm can differ between processes.

color is an integer value that determines in which of the new sub-communicators the current process will fall. All processes of comm for which color has the same numerical value will be part of the same new sub-communicator newcomm.

For example, if you defined color = rank%4 (see Example 4, below), then you would create (globally) four new communicators. Keep in mind that each process will only be seeing the one of these new communicators which it is part of. Put another way, color determines which the "teams" you will create, like the jersey color of football teams.

key will determines how processes are ranked within the new communicators they are part of. If you set key = rank, then the order of ranking (not the ranking itself) in each new communicator newcomm will follow the order of ranking in the original communicator comm. If two or more values of key have the same value, then the process that had the lower rank in comm has the lower rank in newcomm. (See Example 2, below.)

Examples

Here are a few pictorial examples which I duplicate in code below. Example #5 answers your specific question.

Examples of MPI_Comm_split

Code for the Examples

//Compile with mpic++ main.cpp
#include <mpi.h>

int main(int argc, char **argv){
  int world_size, world_rank;

  MPI_Init(&argc, &argv);

  MPI_Comm_size(MPI_COMM_WORLD, &world_size);  //Get the number of processes
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);  //Get the rank of the process

  MPI_Comm comm = MPI_COMM_WORLD;
  MPI_Comm newcomm1, newcomm2, newcomm3, newcomm4, newcomm5;

  //Example 1: Duplicate the existing communicator. The command `MPI_Comm_dup()`
  //           does exactly this.
  MPI_Comm_split(comm, 0, world_rank, &newcomm1);

  //Example 2: Duplicate the existing communicator, but reverse the 
  //           rankings
  MPI_Comm_split(comm, 0, world_size-world_rank, &newcomm2);
  int rank2;                           //Get the rank of the process
  MPI_Comm_rank(newcomm2, &rank2);     //in the new communicator


  //Example 3: Split each process into its own communicator. This is the
  //           equivalent of using `MPI_COMM_SELF` for each process.
  MPI_Comm_split(comm, world_rank, world_rank, &newcomm3);

  //Example 4: Split processes into communicators based on their colouring. Use
  //           their rank in the existing communicator to determine their 
  //           relative rank order in the new communicator.
  int color = world_rank / 4;
  MPI_Comm_split(comm, color, world_rank, &newcomm4);
  int rank4;                           //Get the rank of the process
  MPI_Comm_rank(newcomm2, &rank4);     //in the new communicator

  //Example 5: Group only some of the processes into a new communicator based on
  //a flag.
  int flag = world_rank%2==0;          //An example flag
  MPI_Comm_split(comm, flag?0:MPI_UNDEFINED, world_rank, &newcomm5);

  MPI_Finalize();
}

More Information

  • This page has a nice tutorial on communicators and groups.
  • The SVG file I developed for the examples is available here

Upvotes: 3

Related Questions