MPI_Get doesn't send the correct elements between the buffers of two process

Question

I am trying to create a program that will ultimately be transposing a matrix in MPI so that it can be used in further computations. But right now I am trying to do a simple thing: Root process has a 4x4 matrix "A" which contains elements 0..15 in row-major order. This data is scattered to 2 processes so that each receives one half of the matrix. Process 0 has a 2x4 sub_matrix "a" and receives elements 0..7 and Process 1 gets elements 8..15 in its sub_matrix "a".

My goal is for these processes to swap their a matrices with each other using MPI_Get. Since I was encountering problems, I decided to test a simpler version and simply make process 0 get process 1's "a" matrix, that way, both processes will have the same elements in their respective sub_matrices once I print after the MPI_Get-call and the MPI_fence are called.

Yet the output is erratic, have tried to trouble-shoot for several hours but haven't been able to crack the nut. Would appreciate your help with this.

This is the code below, and the run-command: mpirun -n 2 ./get

Compile: mpicc -std=c99 -g -O3 -o get get.c -lm

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

#define NROWS 4
#define NCOLS 4

int allocate_matrix(int ***M, int ROWS, int COLS) {
  int *p;
  if (NULL == (p = malloc(ROWS * COLS * sizeof(int)))) {
    perror("Couldn't allocate memory for input (p in allocate_matrix)");
    return -1;
  }

  if (NULL == (*M = malloc(ROWS * sizeof(int*)))) {
    perror("Couldn't allocate memory for input (M in allocate_matrix)");
    return -1;
  }

  for(int i = 0; i < ROWS; i++) {
    (*M)[i] = &(p[i * COLS]);
  }
  return 0;
}

int main(int argc, char *argv[])
{
  int rank, nprocs, **A, **a, n_cols, n_rows, block_len;
  MPI_Win win;
  int errs = 0;

  if(rank==0)
    {
      allocate_matrix(&A, NROWS, NCOLS);
      for (int i=0; i<NROWS; i++)
        for (int j=0; j<NCOLS; j++)
          A[i][j] = i*NCOLS + j;
    }

  MPI_Init(&argc,&argv);
  MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
  MPI_Comm_rank(MPI_COMM_WORLD,&rank);

  n_cols=NCOLS; //cols in a sub_matrix
  n_rows=NROWS/nprocs; //rows in a sub_matrix
  block_len = n_cols*n_rows;

  allocate_matrix(&a, n_rows, n_cols);
  for (int i = 0; i <n_rows; i++) 
    for (int j = 0; j < n_cols; j++) 
      a[i][j] = 0;


  MPI_Datatype block_type;
  MPI_Type_vector(n_rows, n_cols, n_cols, MPI_INTEGER, &block_type);
  MPI_Type_commit(&block_type);

  MPI_Scatter(*A, 1, block_type, &(a[0][0]), block_len, MPI_INTEGER, 0, MPI_COMM_WORLD);

  MPI_Barrier(MPI_COMM_WORLD);

  printf("process %d: \n", rank);
  for (int j=0; j<n_rows; j++){
    for (int i=0; i<n_cols; i++){
      printf("%d ",a[j][i]);
    }
    printf("\n");
  }


  if (rank == 0)
    {
      printf("TESTING, before Get a[0][0] %d\n", a[0][0]);
      MPI_Win_create(NULL, 0, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);
      MPI_Win_fence((MPI_MODE_NOPUT | MPI_MODE_NOPRECEDE), win);
      MPI_Get(*a, 8, MPI_INTEGER, 1, 0, 8, MPI_INTEGER, win);
      MPI_Win_fence(MPI_MODE_NOSUCCEED, win);

      printf("TESTING, after Get a[0][0] %d\n", a[0][0]);

      printf("process %d:\n", rank);
      for (int j=0; j<n_rows; j++){
        for (int i=0; i<n_cols; i++){
          printf("%d ", a[j][i]);
        }
        printf("\n");
      }

    }
  else
    { /* rank = 1 */
      MPI_Win_create(a, n_rows*n_cols*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &win);
      MPI_Win_fence((MPI_MODE_NOPUT | MPI_MODE_NOPRECEDE), win);
      MPI_Win_fence(MPI_MODE_NOSUCCEED, win);
    }


  MPI_Type_free(&block_type);
  MPI_Win_free(&win);
  MPI_Finalize();
  return errs;
}

This is the output that I get:

process 0: 
0 1 2 3 
4 5 6 7 

process 1: 
8 9 10 11 
12 13 14 15 

process 0:
1552976336 22007 1552976352 22007 
1552800144 22007 117 0 

But what I want is for the second time I print the matrix from process 0, it should have the same elements as in process 1.

Answer 1

First, I doubt this is really the code you are testing. You are freeing some MPI type variables that are not defined and also rank is uninitialised in

if(rank==0)
  {
    allocate_matrix(&A, NROWS, NCOLS);
    for (int i=0; i<NROWS; i++)
      for (int j=0; j<NCOLS; j++)
        A[i][j] = i*NCOLS + j;
  }

and the code segfaults because A won't get allocated in the root.

Moving this post MPI_Comm_rank(), freeing the correct MPI type variable, and fixing the call to MPI_Win_create in rank 1:

  MPI_Win_create(&a[0][0], n_rows*n_cols*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &win);
  // This -------^^^^^^^^

produces the result you are seeking.

I'd recommend to stick to a single notation for the beginning of the array like &a[0][0] instead of a mixture of *a and &a[0][0]. This will prevent (or at least reduce the occurrence of) similar errors in the future.