CODEWITHSUNDEEP
parallel-processingmpimpi-io
Moody
Moody

Reputation: 1417

Reading text files using MPI-IO?

I have a text file which contains a header with the dimensions of a matrix, and then the matrix. Here's an example for a 3x3 matrix:

3 3
1 56 8
12 3 0
9 44 81

I kept getting garbage values with MPI-IO, and found out it only works with binary, not text files.

I thought I'd read in a character stream and convert to integers, but I'm not sure how to approach this problem since the matrix elements have variable number of digits. I'm really not sure how to approach this?

Upvotes: 0

Views: 1412

Answers (2)

Ed Smith
Ed Smith

Reputation: 13216

Generally you know the type and number of the things written in your binary file (e.g. all integers, 10 int 3 float, etc). You can read as numbers of bytes but MPI binary files are usually read/written as a whole number of types, in your case 9 integers (so number of digits are not important). You open the file and read with something like,

    call MPI_FILE_OPEN(MPI_COMM_WORLD, filename, & 
                   MPI_MODE_RDONLY , MPI_INFO_NULL, fh, ierr)

    bufsize = 3*3
    allocate(buf(bufsize))
    call MPI_FILE_READ_ALL(fh, buf, bufsize, MPI_integer, & 
                           MPI_STATUS_IGNORE, ierr)

For variable matrix sizes, you can use things like MPI_File_get_size to get the size and work out how many elements to read. For mixed data, you can have the first (or last) part of the binary file as a header, which you read first and use to decode the rest of your file. You would still need to know the format of the header and this can be problematic as you break backwards compatibility when you change the code/header format. This is part of the reason for data formats like HDF5 https://support.hdfgroup.org/HDF5/

Upvotes: 1

Rob Latham
Rob Latham

Reputation: 5223

text files are tricky because you need to know 'bytes' not 'numbers'. e.g. 1 1 1 is shorter than 10 15 123355.

now, if your convention says "every number will be zero-padded to 6 digits", then you can have each process read from the (size/nprocs)*rank -th

Or, you'll need an indexer to read the file and record at which offset each row of your matrix starts.

Or as you observed, it all becomes much easier with binary data.

Upvotes: 0

Related Questions