MPI_File_write C

Question

I have to use MPI to compute value of concentration from a 3D matrix.

I divide my matrix in slices depending on the value of the number of proc I will use.

I get smaller 3D matrix which work independently but I want that the values of concentration they compute to be writen in the same file such as the 0 to n-1 lines are writen by proc 0, n to 2n-1 by proc 1,..

I know that i can use an offset in the mpi_file_write_at function to perform such a thing but i don't know how to implement it.

I know that an easy solution would be to ask proc 0 to open the file, write and close it then proc 1 opens.

Don't mind the code in itself as it is not finished

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <mpi.h>

int main(int argc, char *argv[])
{
    char *name = argv[1];

    double h, m, L, T_max, v_x, v_y, v_z, D, r_thresh;
    unsigned int S;

    FILE *file = fopen(name, "rb");


    if(!file)
    {
        fprintf(stderr, "Not able to open the file %s\n", name);
    }
    else
        printf("Able to open the file %s\n", name);

    fscanf(file, "%lf", &h);
    fscanf(file, "%lf", &m);
    fscanf(file, "%lf", &L);
    fscanf(file, "%lf", &T_max);
    fscanf(file, "%lf", &v_x);
    fscanf(file, "%lf", &v_y);
    fscanf(file, "%lf", &v_z);
    fscanf(file, "%lf", &D);
    fscanf(file, "%u", &S);
    fscanf(file, "%lf", &r_thresh);

    fclose(file);

    if(((L/h) % 2) != 0){
        printf("Uncorrect parameters L and h ! \n");
        return -1;
    }
    else{
        unsigned int edge = (L/h);
        unsigned int middle = edge/2;
        unsigned int N = edge+1;
    }
    if(((T_max/m) % 1) != 0){
        printf("Uncorrect parameters T_max and m ! \n");
        return -1;
    }
    else{
        unsigned int t_end = T_max/m;
    }

    unsigned int myrank, nbproc;
    MPI_Init(&argc, &argv) ;
    MPI_Comm_rank( MPI_COMM_WORLD , &myrank );
    MPI_Comm_size( MPI_COMM_WORLD , &nbproc );

    double w = N/nbproc;
    unsigned int width = round(w);
    unsigned int k_min, k_max;
    k_min = myrank*width;
    k_max = (myrank+1)*width-1;
    width = width+2; 
    if(nbproc==1){
        width = N;
    }
    else if(myrank==(nbproc-1)){
        k_max = N-1; 
        width = k_max-k_min+1; 
        width = width+1; 
    }
    else if(myrank==0){
        width = width-1; 
    }



    double ***C ;
    C = malloc(width * sizeof(*C)); 
    for(k=0 ; k < width ; k++){
        C[k] = malloc(N * sizeof(**C));
    }
    for(k=0 ; k < width ; k++){
        for(j=0 ; j < N ; j++){
            C[k][j] = malloc(N * sizeof(***C));
        }
    }


    unsigned int i,j,k,n;
    for(k=0 ; k<width ; k++){
        for(j=0 ; j<N ; j++){
            for(i=0 ; i<N ; i++){
                C[k][j][i]=0;
            }
        }
    }
    if(middle>=k_min && middle=<k_max){
        C[middle-k_min][middle][middle] = 1;
    }


    if(myrank==0 || nbproc==1){
        k_min = k_min +1 ;
    }
    if(myrank==(nbproc-1) || nbproc==1){
        k_max = k_max -1 ;
    }





    for(n=1 ; n<T_max ; n++){

        char *c = "c_";
        char buffer[10];
        itoa(n,buffer,2);
        strcat(c,buffer);

        // I want to write some value stored in a buffer now
        // computed by different process in the same file 
        // MPI_File_write_at ???


    }

    for(k=0 ; k < width ; k++){
        for(j=0 ; j < N ; j++){
            free(C[k][j]);
            }
        free(C[k]);
    }
    free(C);

    return 0;
}

Answer 1

Each process in an MPI program has a local address space. File I/O changes that: a file is a globally accessible linear stream of bytes. You need to think about where in the global space each process is writing. A couple ways to do this:

• sometimes you can just have each process write to a corresponding chunk of the file. rank zero writes to the first megabyte, rank 1 to the 2nd, and so on. Might be too simplistic for your 3d array.

• For more sophisticated decomposition, you set a "file view": you construct an MPI datatype describing the layout of data on disk. in your case, MPI_TYPE_CREATE_SUBARRAY is probably what you want: a 3d global matrix and each process contributes some subaray to the global picture.