CODEWITHSUNDEEP
cmpinonblocking
Jackie
Jackie

Reputation: 13

A loop of MPI non-blocking communication fails over a certain number of nodes

I am writing a piece of code which could pass data around the nodes in a circle, just like rank0->rank1 rank1->rank2 rank2->rank0.

To test the bandwidth, a for loop is added to do the non-blocking communication hundreds of time.

When I use 5 or less nodes, the code works. But in the cases of more than 5 nodes, it fail.

[compute-0-10:21508] Signal: Segmentation fault (11) [compute-0-10:21508] Signal code: (128) [compute-0-10:21508] Failing at address: (nil)

However, if I comment out the loop lines, only sending and receiving the data once, the code works regardless of the number of nodes. I have really no idea where goes wrong.

The following is the code. 

int rank, npes;
MPI_Comm_size(MPI_COMM_WORLD,&npes);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);

float* recv_buffer_last = (float*)malloc(sizeof(float)*YMAX*FMAX);
float* recv_buffer_next = (float*)malloc(sizeof(float)*YMAX*FMAX);
float* send_buffer_next = (float*)malloc(sizeof(float)*YMAX*FMAX);
float* send_buffer_last = (float*)malloc(sizeof(float)*YMAX*FMAX);


if(npes >1)
    {
      MPI_Status reqstat;
      MPI_Request send_request;
      MPI_Request recv_request;

      for(int loop = 0; loop < 100; loop++) //if no loop here,code always works.
        {
          MPI_Irecv(recv_buffer_last,YMAX*FMAX,MPI_FLOAT,(rank == 0)?(npes - 1):(rank - 1),100,MPI_COMM_WORLD,&recv_request);
          MPI_Irecv(recv_buffer_next,YMAX*FMAX,MPI_FLOAT,(rank == npes-1)?0:rank+1,1000,MPI_COMM_WORLD,&recv_request);
          MPI_Isend(send_buffer_next,YMAX*FMAX,MPI_FLOAT,(rank == npes -1)?0:rank+1,100,MPI_COMM_WORLD,&send_request);
          MPI_Isend(send_buffer_last,YMAX*FMAX,MPI_FLOAT,(rank ==0)?(npes - 1):(rank - 1),1000,MPI_COMM_WORLD,&send_request);
          MPI_Waitall(1,&recv_request,&reqstat);
          MPI_Waitall(1,&send_request,&reqstat);

        }
    }
  else
    {
      memcpy(recv_buffer_last,send_buffer_next,sizeof(float)*YMAX*FMAX);
      memcpy(recv_buffer_next,send_buffer_last,sizeof(float)*YMAX*FMAX);
    }

UPDATES:

Thank you everyone for pointing out the error of MPI_Waitall, but making that modification doesn't solve my problem.

After playing around my code, I have found that changing dynamic array recv_buffer and send_buffer to static ones making the code works perfectly. 

int rank, npes;
MPI_Comm_size(MPI_COMM_WORLD,&npes);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);

/*
float* recv_buffer_last = (float*)malloc(sizeof(float)*YMAX*FMAX);
float* recv_buffer_next = (float*)malloc(sizeof(float)*YMAX*FMAX);
float* send_buffer_next = (float*)malloc(sizeof(float)*YMAX*FMAX);
float* send_buffer_last = (float*)malloc(sizeof(float)*YMAX*FMAX);
*/

float recv_buffer_last[FMAX*YMAX];
float recv_buffer_next[FMAX*YMAX];
float send_buffer_last[FMAX*YMAX];
float send_buffer_next[FMAX*YMAX];

int prev = (rank+npes-1)%npes;
int next = (rank+1)%npes;

if(npes >1)
    {
      MPI_Request requests[4];
      for(int loop = 0; loop < 100; loop++)
       {
        MPI_Irecv(recv_buffer_last,YMAX*FMAX,MPI_FLOAT,prev,100,MPI_COMM_WORLD,&requests[0]);
        MPI_Irecv(recv_buffer_next,YMAX*FMAX,MPI_FLOAT,next,1000,MPI_COMM_WORLD,&requests[1]);
        MPI_Isend(send_buffer_next,YMAX*FMAX,MPI_FLOAT,next,100,MPI_COMM_WORLD,&requests[2]);
        MPI_Isend(send_buffer_last,YMAX*FMAX,MPI_FLOAT,prev,1000,MPI_COMM_WORLD,&requests[3]);
        MPI_Waitall(4,requests,MPI_STATUSES_IGNORE);


        }
    }
  else
    {
      memcpy(recv_buffer_last,send_buffer_next,sizeof(float)*YMAX*FMAX);
      memcpy(recv_buffer_next,send_buffer_last,sizeof(float)*YMAX*FMAX);
    }

I am wondering what makes the difference so that I can avoid the similar problems in the future.

Upvotes: 0

Views: 406

Answers (1)

Gilles
Gilles

Reputation: 9509

I'm pretty sure the source of your problem is that you have dangling requests that you do not wait for at the end of your communications, since you have 2 MPI_Irecv() and 2 MPI_Isend() per loop, so 4 requests posted, but only 2 requests waited for. This means that internally, the MPI library will allocate resources for these requests, which will never be freed, leading to hitting some internal limit and to the error you encountered.

Here is a example of what your code could look like:

int rank, npes;
MPI_Comm_size( MPI_COMM_WORLD, &npes );
MPI_Comm_rank( MPI_COMM_WORLD, &rank );

int len = YMAX*FMAX;
float *recv_buffer_last = (float*) malloc( sizeof( float ) * len );
float *recv_buffer_next = (float*) malloc( sizeof( float ) * len );
float *send_buffer_next = (float*) malloc( sizeof( float ) * len );
float *send_buffer_last = (float*) malloc( sizeof( float ) * len );

if ( npes > 1 ) {
    MPI_Request requests[4];
    int prev = ( rank + npes - 1 ) % npes;
    int next = ( rank + 1 ) % npes;
    for ( int loop = 0; loop < 100; loop++ ) {
        MPI_Irecv( recv_buffer_last, len, MPI_FLOAT, prev,  100, MPI_COMM_WORLD, &requests[0] );
        MPI_Irecv( recv_buffer_next, len, MPI_FLOAT, next, 1000, MPI_COMM_WORLD, &requests[1] );
        MPI_Isend( send_buffer_next, len, MPI_FLOAT, next,  100, MPI_COMM_WORLD, &requests[2] );
        MPI_Isend( send_buffer_last, len, MPI_FLOAT, prev, 1000, MPI_COMM_WORLD, &requests[3] );
        MPI_Waitall( 4, requests, MPI_STATUSES_IGNORE );
    }
}
else {
    memcpy( recv_buffer_last, send_buffer_next, sizeof( float ) * len );
    memcpy( recv_buffer_next, send_buffer_last, sizeof( float ) * len);
}

Upvotes: 1

Related Questions