ulysses
ulysses

Reputation: 41

How do i declare the integer (MPI_OFFSET_KIND) in fortran, required by MPI_File_set_view, to prevent overflow when writing large data?

Intro:

I am trying to write a large set of data to a single file using MPI IO using the code below. The problem i encounter is, that i get an integer overflow (variable disp) and thus the MPI IO does not work properly. The reason for this is, i think, the declaration of the variable disp (integer (kind=MPI_OFFSET_KIND) :: disp = 0) in the subroutine write_to_file(...). Since for the process with the highest rank disp overflows.

Question:

Can I somehow define disp as kind=MPI_OFFSET_KIND but with higher range? I did not find a solution for that, except writing to multiple files, but I would prefer writing into a single file.

Some context:

The code is just a part of an code, which i use to output (and read; but I cut that part from the code example to make it easier) scalar (ivar = 1), vector(ivar=3) or tensor(ivar=3,6 or 9) values into binary files. The size of the 3D grid is defined by imax, jmax and kmax, where kmax is decomposed by Px processes into Mk. Lately the 3D grid grew to a size where i encountered the described problem.

Code Example: MPI_IO_LargeFile.f90

"""

PROGRAM MPI_IO_LargeFile

use MPI

implicit none

integer rank, ierr, Px
integer i, j, k, cnt
integer imax, jmax, kmax, Mk
integer status(MPI_STATUS_SIZE)
integer ivars;
real*4, dimension(:,:,:,:), allocatable :: outarr, dataarr
call MPI_Init(ierr)
call MPI_Comm_size(MPI_COMM_WORLD, Px, ierr)
call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)

imax = 256
jmax = 512
kmax = 1024
Mk = kmax / Px

if (rank < 1) print *, 'Preparing dataarr'
ivars = 6
allocate(dataarr(ivars,imax,jmax,Mk))
call RANDOM_NUMBER(dataarr)

! Output of Small File
if (rank < 1) print *, 'Output of SmallFile.txt'
ivars = 3
allocate(outarr(ivars,imax,jmax,Mk))
outarr(:,:,:,:) = dataarr(1:3,:,:,:)
call write_to_file(rank, 'SmallFile.txt', outarr)
deallocate(outarr)

! Output of Large File
if (rank < 1) print *, 'Output of LargeFile.txt'
ivars = 6
allocate(outarr(ivars,imax,jmax,Mk))
outarr(:,:,:,:) = dataarr(1:6,:,:,:)
call write_to_file(rank, 'LargeFile.txt', outarr)

deallocate(outarr)
deallocate(dataarr)
call MPI_Finalize(ierr)

CONTAINS

subroutine write_to_file(myrank, filename, arr)
  implicit none

  integer, intent(in) :: myrank
  integer :: ierr, file, varsize
  character(len=*), intent(in):: filename
  real*4, dimension(:,:,:,:), allocatable, intent(inout) :: arr
  **integer (kind=MPI_OFFSET_KIND) :: disp = 0**

  varsize = size(arr)
  disp = myrank * varsize * 4

  **write(*,*) rank, varsize, disp**

  call MPI_File_open(MPI_COMM_WORLD, filename, &
    & MPI_MODE_WRONLY + MPI_MODE_CREATE, &
    & MPI_INFO_NULL, file, ierr )
  call MPI_File_set_view(file, disp, MPI_REAL4, &
    & MPI_REAL4, 'native', MPI_INFO_NULL, ierr)
  call MPI_File_write(file, arr, varsize, &
    & MPI_REAL4, MPI_STATUS_IGNORE, ierr)
  call MPI_FILE_CLOSE(file, ierr)

end subroutine write_to_file

END PROGRAM MPI_IO_LargeFile

"""

Output of Code: MPI_IO_LargeFile.f90

mpif90 MPI_IO_LargeFile.f90 -o MPI_IO_LargeFile
mpirun -np 4 MPI_IO_LargeFile
 Preparing dataarr
 Output of SmallFile.txt
           2   100663296            805306368
           1   100663296            402653184
           3   100663296           1207959552
           0   100663296                    0
 Output of LargeFile.txt
           1   201326592            805306368
           0   201326592                    0
           2   201326592           1610612736
           3   201326592          -1879048192
mca_fbtl_posix_pwritev: error in writev:Invalid argument
mca_fbtl_posix_pwritev: error in writev:Invalid argument

Upvotes: 2

Views: 428

Answers (1)

ulysses
ulysses

Reputation: 41

The Problem is that the multiplication in

disp = myrank * varsize * 4

overflowed, since each variable was declared as integer.

One solution provided by @Gilles (in the comments of the question) was simply to change this line to

disp = myrank * size(arr, kind=MPI_OFFSET_KIND) * 4

Using size(arr, kind=MPI_OFFSET_KIND) converts the solution into an integer of kind=MPI_OFFSET_KIND, which solves the overflow problem.

Thank you for your help.

Upvotes: 2

Related Questions