CODEWITHSUNDEEP
c++mpi
Shibli
Shibli

Reputation: 6149

Same send and receive buffers in MPI

In my code, each process works on certain portion of an array. I want each process to send the portion it worked on to other processes and receive other portions from other processes. For this I used MPI_Allgatherv but I kept send and receive buffers the same:

MPI_Allgatherv (&vel[0],  localSizesFaceV[rank], MPI_DOUBLE, &vel[0],  localSizesFaceV, displsFaceV, MPI_DOUBLE, MPI_COMM_WORLD);

I used this function before for other purposes with different send and receive buffers and it worked. That is why I am sure there is no problem with other parameters.

In the case of 2 processes, one of the processes does not return. When I copied send buffer to another std::vector

vector <double> vel2;
vel2 = vel;

and used vel2 as send buffer then all processes returned. Why?

Upvotes: 4

Views: 2669

Answers (2)

Gilles
Gilles

Reputation: 9489

Generally speaking, MPI requires that the argument is not aliased. This is explicitly mentioned chapter 2.3 of the current standard.

Unless specified otherwise, an argument of type OUT or type INOUT cannot be aliased with any other argument passed to an MPI procedure.

This explains why your code has problems. However, there is the possibility to solve your issue very easily, without having to explicitly copy your buffer: the MPI_IN_PLACE keyword. It specifies that the communication will be done "in-place" using the output buffer as an input buffer too wherever relevant.

Your code would become:

MPI_Allgatherv( MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, &vel[0],  localSizesFaceV, displsFaceV, MPI_DOUBLE, MPI_COMM_WORLD);

NB: the actual type to use for the send buffer is irrelevant. You can keep MPI_DOUBLE if you want to, but I tend to prefer using MPI_DATATYPE_NULL to make clear that the parameter is ignored.

Upvotes: 8

hauron
hauron

Reputation: 4668

From the spec http://www.mpich.org/static/docs/v3.1/www3/MPI_Allgatherv.html

int MPI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype,
               void *recvbuf, const int *recvcounts, const int *displs,
               MPI_Datatype recvtype, MPI_Comm comm)

sendbuf is const, yet you also pass a pointer allowing to change sendbuf underlying memory via recvbuf. This looks risky, but I do not know MPI well enough to be sure.

I'd also assume there's a possible race condition on the buffer memory, and if so - this is undefined behaviour.

Upvotes: 3

Related Questions