CODEWITHSUNDEEP
cmpi
bob.sacamento
bob.sacamento

Reputation: 6651

Gathering in MPI, but not with MPI_Gather

I need to do something like an MPI_Gather, but I need to store the gathered data in an order that depends on something different from process rank. I thought I could do something like this:

int i;
int gathervals[10];
int my_val, my_place_in_storage;

/* init MPI etc ... then ... */

my_val = some_fcn_of_rank(my_rank, ...)
my_place_in_storage = some_other_fcn(my_val, my_rank, ...)

if (my_rank == 0){
/* my_val on proc 0 does go in gathervals[0], which simplifies
   things */
   gathervals[0] = my_val;
   for (i=1; i<num_procs; i++){
      MPI_Recv(&gathervals[i], 1, MPI_INT, MPI_ANY_SOURCE,
               i, MPI_COMM_WORLD, stat_mpi);
   }
}else{
   MPI_Send(&mv_val, 1, MPI_INT, 0, my_place_in_storage,
            MPI_COMM_WORLD);
}

My thinking is that proc 0 will start the loop and wait until a Send is posted from the proc whose my_place_in_storage is 1, then proc 0 will receive that message and put the value into gathervals[1]. Then it will iterate and wait until it sees a post from the proc whose my_place_in_storage is 2, etc.

Should this work? I'm getting a seg fault on proc 0 and I'm trying to figure out why. I thought this block of code, which might be a little unorthodox, would be the place to start. I have verified the correctness of all the my_place_in_storage values.

Upvotes: 1

Views: 881

Answers (2)

Zulan
Zulan

Reputation: 22670

Yes, this should work, but you there is a better alternative. Use MPI_Gather to collect my_place_in_storage and use it as displacement array for MPI_Gatherv. Something along the lines:

 if (my_rank == 0) {
     int displacements[10];
     int recvcounts[10] = {1,1,1,1,1,1,1,1,1,1};
     MPI_Gather(&my_place_in_storage, 1, MPI_INT,
                displacements, 1, MPI_INT, 0, MPI_COMM_WORLD); 
     MPI_Gatherv(&my_val, 1, MPI_INT, gathervals, recvcounts,
                 displacements, MPI_INT, 0, MPI_COMM_WORLD);
 } else {
     MPI_Gather(&my_place_in_storage, 1, MPI_INT,
                NULL, 1, MPI_INT, 0, MPI_COMM_WORLD); 
     MPI_Gatherv(&my_val, 1, MPI_INT, NULL, 1, NULL,
                 MPI_INT, 0, MPI_COMM_WORLD);
 }

In general, it is better to use a collective instead of multiple point to point messages. This is more scalable and allows more optimizations by the MPI implementation.

Upvotes: 4

David Henty
David Henty

Reputation: 1764

The other option is to use MPI_Comm_split to create a new communicator which contains all the same processes as MPI_COMM_WORLD but permuted into a new order based on "my_place_in_storage". You could do this if everyone supplied the same value of "color" so they all belonged to the same communicator, but used "my_place_in_storage" as the "key".

Communicator creation has some overhead but this might be more efficient overall than using two gathers if you do this operation many times with the same values of "my_place_in_storage". Note that, in the new communicator, you would also just use gather which should also be faster than gatherv.

Upvotes: 5

Related Questions