Reputation: 31
I've been trying out the partitioned communication that was added in openmpi 5. I am aware that "The current implementation is an early prototype and is not fully compliant with the MPI-4.0 specification" as it says in the docs. However the deadlock issue that I have encountered doesn't seem to pertain to that (at least as far as I can tell).
The issue is that if I try to engage in partitioned communication with more than one partner the "second" communication does not complete/match. The first two processes are blocking at the barrier while the last process waits on his request to complete. However I can not tell why that would be the case, since the tags and ranks match in the MPI_Psend_init
/MPI_Precv_init
match. Am I using the feature incorrectly? I could not find anything about this in the openmpi 5 documentation. I have also not tried other implementations of mpi.
#include <mpi.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
int main(int argc, char *argv[]) {
MPI_Init(&argc, &argv);
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (size != 3) {
if (rank == 0)
printf("Need 3 workers, you have %d\n", size);
return -1;
}
/*
Topology
0 <-> 1 <-> 2
*/
const int partitions = 4;
const int count = 16;
int buffer[partitions * count];
MPI_Request request[2];
memset(buffer, 0, sizeof(int) * partitions * count);
if (rank == 1) {
for (int i = 0; i < partitions * count; i++) {
buffer[i] = i;
}
}
int nor = 0;
if (rank == 0) {
MPI_Precv_init(buffer, partitions, count, MPI_INT, 1, 0, MPI_COMM_WORLD, MPI_INFO_NULL, &request[nor++]);
} else if (rank == 1) {
MPI_Psend_init(buffer, partitions, count, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_INFO_NULL, &request[nor++]);
MPI_Psend_init(buffer, partitions, count, MPI_INT, 2, 0, MPI_COMM_WORLD, MPI_INFO_NULL, &request[nor++]);
} else if (rank == 2) {
MPI_Precv_init(buffer, partitions, count, MPI_INT, 1, 0, MPI_COMM_WORLD, MPI_INFO_NULL, &request[nor++]);
}
MPI_Startall(nor, request);
if (rank == 1) {
MPI_Pready_range(0, partitions-1, request[0]);
MPI_Pready_range(0, partitions-1, request[1]);
}
MPI_Waitall(nor, request, MPI_STATUS_IGNORE);
MPI_Barrier(MPI_COMM_WORLD);
for (int i = 0; i < nor; i++) {
MPI_Request_free(request + i);
}
MPI_Finalize();
return 0;
}
I have tried to debug the message queue using linaro's ddt. This only shows that rank 1 sent it's messages to rank 2 but rank 2's receive requests are not being matched (even though they have the same tags as rank 1's sends), whereas rank 0's requests were matched. deadlock state in barrier and wait calls
Upvotes: 3
Views: 38