Persistent communication in MPI - odd behaviour

Question

I am solving the coarsest grid of parallel geometric multi grid using jacobi iterations and using Non-blocking calls MPI_Isend() and MPI_Irecv(). There are no problems in this. As soon as I replace the non-blocking communications with persistent communications - the results stop converging at this level and program goes into an infinite loop. The calls MPI_Startall() and MPI_Waitall() always return MPI_SUCCESS. Has anyone faced this problem before ? Please advise.

 Coarsest_grid_solve()
{
MPI_Recv_init(&e_c_old[0][1][1], 1, x_subarray_c, X_DOWN, 10, new_comm, &recv[0]);
MPI_Recv_init(&e_c_old[PXC+1][1][1], 1, x_subarray_c, X_UP, 20, new_comm, &recv[1]);
MPI_Recv_init(&e_c_old[1][PYC+1][1], 1, y_subarray_c, Y_RIGHT, 30, new_comm, &recv[2]);
MPI_Recv_init(&e_c_old[1][0][1], 1, y_subarray_c, Y_LEFT, 40, new_comm, &recv[3]); 
MPI_Recv_init(&e_c_old[1][1][PZC+1], 1, z_subarray_c, Z_AWAY_U, 50, new_comm, &recv[4]);
MPI_Recv_init(&e_c_old[1][1][0], 1, z_subarray_c, Z_TOWARDS_U, 60, new_comm, &recv[5]);

MPI_Send_init(&e_c_old[PXC][1][1], 1, x_subarray_c, X_UP, 10, new_comm, &send[0]);
MPI_Send_init(&e_c_old[1][1][1], 1, x_subarray_c, X_DOWN, 20, new_comm, &send[1]);
MPI_Send_init(&e_c_old[1][1][1], 1, y_subarray_c, Y_LEFT, 30, new_comm, &send[2]);
MPI_Send_init(&e_c_old[1][PYC][1], 1, y_subarray_c, Y_RIGHT, 40, new_comm, &send[3]);
MPI_Send_init(&e_c_old[1][1][1], 1, z_subarray_c, Z_TOWARDS_U, 50, new_comm, &send[4]);
MPI_Send_init(&e_c_old[1][1][PZC], 1, z_subarray_c, Z_AWAY_U, 60, new_comm, &send[5]);  

while(rk_global/r0_global > TOL_CNORM)              
{
coarse_iterations++ ;           

err = MPI_Startall(6,recv);
if(err == MPI_SUCCESS)
    printf("success");

err = MPI_Startall(6,send);
if(err == MPI_SUCCESS)
    printf("success");

err = MPI_Waitall(6, send, MPI_STATUSES_IGNORE);
if(err == MPI_SUCCESS)
    printf("success");

err = MPI_Waitall(6, recv, MPI_STATUSES_IGNORE);
if(err == MPI_SUCCESS)
    printf("success");

//do work here

if(coarse_iterations == 1)
{
    update_neumann_c(e_c_old, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U);
    residual_coarsest(e_c_old, rho_c, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U, hc, rho_temp); 
    r0_local = residual_norm(rho_temp, PXC, PYC, PZC); 
    start_allred = MPI_Wtime();
    MPI_Allreduce(&r0_local, &r0_global, 1, MPI_DOUBLE, MPI_SUM, new_comm); 
    end_allred = MPI_Wtime(); 
    r0_global = r0_global/( (PXC*dims0) * (PYC*dims1) * (PZC*dims2) ); 
    if(rank == 0)
        printf("\nGlobal residual norm is = %f", r0_global);
    rk_global = r0_global;
}           
else
{
    update_neumann_c(e_c_old, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U);
    residual_coarsest(e_c_old, rho_c, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U, hc, rho_temp); 
    rk_local = residual_norm(rho_temp, PXC, PYC, PZC);
    start_allred = MPI_Wtime(); 
    MPI_Allreduce(&rk_local, &rk_global, 1, MPI_DOUBLE, MPI_SUM, new_comm);
    end_allred = MPI_Wtime();  
    rk_global = rk_global/( (PXC*dims0) * (PYC*dims1) * (PZC*dims2) );
    if(rank == 0)
        printf("\nGlobal residual norm is = %f", rk_global);
}

//do dependent work and exchange matrices
}//while loop ends

for(i = 0; i <= 5 ; i++)
{
    MPI_Request_free(&send[i]);
    MPI_Request_free(&recv[i]);
}
}//End coarsest grid solve

Note: Strangely the ghost data becomes zero on alternate iterations. (Just found out - don't know why).

Answer 1

When we make a persistent handle for communication, we point to a specific piece of memory that we want to transfer to some other process. Now in Jacobi iterations we need to exchange pointers at the end of the iteration to make the old matrix point to the new updated matrix. Thus, the memory location pointed to by the pointer changes. Thus, the original locations are exchanged. The way around is to define two persistent communication handles. On odd iterations use the first handle and on even iterations use the other handle i.e. alternate them. This solved my problem. This problem also widened my understanding of persistent communications in MPI.