CODEWITHSUNDEEP
cparallel-processingmpi
Deniz Ünal
Deniz Ünal

Reputation: 1

How to read different interval of lines from a text file in different processes using MPI in C

I am trying to portion out 1 million lines of float numbers to 16 different processes. For example, process 0 needs to read between lines 1-62500 and process 1 needs to read between lines 62501-125000 etc.

I have tried the following code, but every process reads the lines between 1-62500. How can I change the line interval for each process?

MPI_Init(NULL, NULL);

n=1000000/numberOfProcesses;

FILE *myFile;

myFile = fopen("input.txt","r");
i=0;
k = n+1;
while(k--){
    fscanf(myFile,"%f",&input[i]);
    i++;
}
fclose(myFile);

MPI_Finalize();

Upvotes: 0

Views: 229

Answers (1)

j23
j23

Reputation: 3530

Assuming numbeOfProcesses=4 and numberOfLines=16

//so new n will be 4
//n=1000000/numberOfProcesses;
n=numberOfLines/numbeOfProcesses
FILE *myFile;

myFile = fopen("input.txt","r");
i=0;
k = n+1 //(5)

From your program, all processes will read the file from the same location or offset. What you need to do is to make each process read from their own specific line or offset. For example, rank 0 should read from 0, rank 1 from n, rank 2 from 2*n etc. Pass this as parameter to fseek.

n=numberOfLines/numbeOfProcesses
MPI_Comm_rank(MPI_COMM_WORLD,&rank)
file_start= n*rank
fseek(myfile, file_start, SEEK_SET);

fseek will go the offset (file_start) of the file. Then file_start will be 4 for rank 0, 8 for rank 1 etc...

Also while loop should be modified accordingly.

As @Gilles pointed out in comments, here we are explicitly assuming the number of lines in the file. This can lead to many issues.

To get scalability and parallel performance benefits, it is better to use MPI IO, which offers great features for parallel file operations. MPI IO is developed for this kind of usecases.

Upvotes: 1

Related Questions