CODEWITHSUNDEEP
cmpi
user3124318
user3124318

Reputation: 13

Joining output binary files from MPI simulation

I have 64 output binary files from an MPI simulation using a C code.

The files correspond to the output of 64 processes. What would be a way to join all those files into a single file, perhaps using a C script?

Upvotes: 0

Views: 105

Answers (2)

kag0
kag0

Reputation: 6054

Since all the output files are fairly small and the same size, it would be easy to use MPI_Gather to assemble one large binary array on one node which could then be written to a file. If allocating a large array is an issue, you could simply use MPI_ISend and MPI_Recv to write to the file one piece at at time.

Obviously this is a pretty primitive solution, but it is also very straightforward, foolproof and really won't take notably longer (assuming you're doing all this at the end of your simulation).

Upvotes: 0

Rob Latham
Rob Latham

Reputation: 5223

Since this was tagged MPI, I'll offer an MPI solution, though it might not be something the questioner can do.

If you are able to modify the simulation, why not adopt an MPI-IO approach? Even better, look into HDF5 or Parallel-NetCDF and get a self-describing file format, platform portability, and a host of analysis and vis tools that already understand your file format.

But no matter which approach you take, the general idea is to use MPI to describe which part of each file belongs to each process. The easiest example is if each process contributes to a 1D array. then for a logically global array of N items, each process contributes 1/N items at offset "myrank/N"

Upvotes: 2

Related Questions