CODEWITHSUNDEEP
memoryfortranopenmp
ftiaronsem
ftiaronsem

Reputation: 1584

690mb memory overhead for openmp program compiled with ifort

I was running some tests using openmp and fortran and came to realize that a binary compiled with ifort 15 (15.0.0 20140723) has 690MB of virtual memory overhead.

My sample program is:

program sharedmemtest
  use omp_lib
  implicit none
  integer :: nroot1
  integer, parameter :: dp = selected_real_kind(14,200)
  real(dp),allocatable :: matrix_elementsy(:,:,:,:)

  !$OMP PARALLEL NUM_THREADS(10) SHARED(matrix_elementsy)                                                                                                                           
  nroot1=2

  if (OMP_GET_THREAD_NUM() == 0) then
    allocate(matrix_elementsy(nroot1,nroot1,nroot1,nroot1))
    print *, "after allocation"
    read(*,*)
  end if
  !$OMP BARRIER                                                                                                                                                                     

  !$OMP END PARALLEL                                                                                                                                                                
end program

running 

ifort -openmp test_openmp_minimal.f90 && ./a.out

shows a memory usage of

50694 user    20   0  694m 8516 1340 S  0.0  0.0   0:03.58 a.out

in top. Running

gfortran -fopenmp test_openmp_minimal.f90 && ./a.out

shows a memory usage of

50802 user    20   0 36616  956  740 S  0.0  0.0   0:00.98 a.out

Where is the 690MB of overhead coming from when compiling with ifort? Am I doing something wrong? Or is this a bug in ifort?

For completeness: This is a minimal example taken from a much larger program. I am using gfortran 4.4 (4.4.7 20120313).

I appreciate all comments and ideas.

Upvotes: 1

Views: 203

Answers (1)

Jeff Hammond
Jeff Hammond

Reputation: 5652

I don't believe top is reliable here. I do not see any evidence that the binary created from your test allocates anywhere near that much memory.

Below I have shown the result of generating the binary normally, with the Intel libraries linked statically and with everything linked statically. The static binary is in the ballpark of 2-3 megabytes.

It is possible that OpenMP thread stacks, which I believe are allocated from the heap, could be the source of the addition virtual memory here. Can you try this test with OMP_STACKSIZE=4K? I think the default is a few megabytes.

Dynamic Executable

jhammond@cori11:/tmp> ifort -O3 -qopenmp smt.f90 -o smt
jhammond@cori11:/tmp> size smt
   text    data     bss     dec     hex filename
 748065   13984  296024 1058073  102519 smt
jhammond@cori11:/tmp> ldd smt
    linux-vdso.so.1 =>  (0x00002aaaaaaab000)
    libm.so.6 => /lib64/libm.so.6 (0x00002aaaaab0c000)
    libiomp5.so => /opt/intel/parallel_studio_xe_2016.0.047/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64/libiomp5.so (0x00002aaaaad86000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaab0c7000)
    libc.so.6 => /lib64/libc.so.6 (0x00002aaaab2e4000)
    libgcc_s.so.1 => /opt/gcc/5.1.0/snos/lib64/libgcc_s.so.1 (0x00002aaaab661000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaab878000)
    /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

Dynamic Executable with Static Intel

jhammond@cori11:/tmp> ifort -O3 -qopenmp smt.f90 -static-intel -o smt
jhammond@cori11:/tmp> size smt
   text    data     bss     dec     hex filename
1608953   41420  457016 2107389  2027fd smt
jhammond@cori11:/tmp> ls -l smt
-rwxr-x--- 1 jhammond jhammond 1872489 Jan 12 05:51 smt

Static Executable

jhammond@cori11:/tmp> ifort -O3 -qopenmp smt.f90 -static -o smt
jhammond@cori11:/tmp> size smt
   text    data     bss     dec     hex filename
2262019   43120  487320 2792459  2a9c0b smt
jhammond@cori11:/tmp> ldd smt
    not a dynamic executable

Upvotes: 3

Related Questions