CODEWITHSUNDEEP
c++mpi
Suslik
Suslik

Reputation: 1071

MPI receiving data from an unknown number of ranks

I have a list of indices for which I do not know their corresponding entries in a vector, because the vector is distributed among the ranks. I have to send these indices to the ranks in charge to get the data.

On the other hand "my" rank also get lists of indices from an unknown number of ranks. After receiving the list, "my" rank has to send the corresponding data to this requesting ranks.

I think I have to work with a mixture of MPI_Probe and MPI_Gather. But at the moment I cannot see how to receive lists from an unknown number of ranks.

I think it has to look like this, but how can I receive the data from a bigger unknown number of rank? Or do I have to loop over all possible ranks, that could send me something?

 MPI_Status status;
 int nbytes; 
 std::vector<Size> indices;
 MPI_Probe(MPI_ANY_SOURCE,MPI_ANY_TAG, comm, &status);
 MPI_Get_count(&status,MPI_UINT64_T, &nbytes);

 if(nbytes!=MPI_UNDEFINED){
     indices.reserve(nbytes);
     MPI_Recv(&indices[0],nbytes,MPI_UINT64_T,status.SOURCE,status.TAG,comm,&status);
 }

Upvotes: 0

Views: 244

Answers (1)

Matthieu Brucher
Matthieu Brucher

Reputation: 22023

This resembles a lot what I did a few years ago for parallel I/O.

One option:

  • From all senders, get the size that you need to send to each other rank
  • Send the sizes (Allgather if all ranks can be senders, otherwise sends/receives)
  • Does a (all)gatherv that will retrieve the size on each receiver

You can use non blocking send/receives as well as gatherv (MPI3) and this scales well (depending ont he hardware) to 500 cores for 8 senders.

The way we did it was to go through the vector by chunk of several MB and send the data in chunks. Of course, the bigger the chunks the better, but also the more memory you need on each sender ranks to hold the data.

Upvotes: 1

Related Questions