How can I get my Fortran program to use a certain amount of RAM?

Question

I am trying to write a Fortran program which will eat up a lot of memory (for the reasoning behind this, please see the note at the end of this question). I am doing this by allocating a 3 dimensional array of size (n,n,n) and then deallocating it - continually increasing n until I run out of memory (this should happen when ~16 GB of memory is used). Unfortunately, it seems as if my program is running out of memory long before I see the system resources get up to 16 GB.

Here is my sample code:

 1 program fill_mem
 2   implicit none
 3   integer, parameter :: ikind = selected_int_kind(8)
 4   integer, parameter :: rkind = 8
 5
 6   integer(kind = ikind) :: nfiles = 100
 7   integer(kind = ikind) :: n = 1200
 8   integer(kind = ikind) :: i, nn
 9
10   real(kind = rkind), allocatable :: real_arr(:,:,:)
11
12   character(500) :: sysline
13
14
15   call system('echo ''***no_allocation***'' > outfile')
16   call system('ps aux | grep fill_mem.exe >> outfile')
17   !call system('smem | grep fill_mem.exe >> sm.out')
18   allocate(real_arr(n, n, n))
19
20   nn = 100000
21   do i = 1,nn
22     deallocate(real_arr)
23     n = n + 10
24     print*, 'n = ', n
25     allocate(real_arr(n, n, n))
26     call system('echo ''*************'' >> outfile')
27     write(sysline, *) 'allocation', i, '... n = ', n
28
29     write(*, '(f10.5, a)') 100.0*real(i)/real(nn), '%'
30
31     call system(trim(adjustl('echo '//sysline//'>> outfile')))
32     call system('ps aux | grep fill_mem.exe >> outfile')
33   enddo
34
35 end program fill_mem

and here is the sample output:

 1 ***no_allocation***
 2 1000     12350  0.0  0.0  12780   760 pts/1    S+   13:32   0:00 ./fill_mem.exe
 3 1000     12352  0.0  0.0   4400   616 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
 4 1000     12354  0.0  0.0   9384   920 pts/1    S+   13:32   0:00 grep fill_mem.exe
 5 *************
 6 allocation 1 ... n = 1210
 7 1000     12350  0.0  0.0 13853104 796 pts/1    S+   13:32   0:00 ./fill_mem.exe
 8 1000     12357  0.0  0.0   4400   616 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
 9 1000     12359  0.0  0.0   9384   920 pts/1    S+   13:32   0:00 grep fill_mem.exe
10 *************
11 allocation 2 ... n = 1220
12 1000     12350  0.0  0.0 14199096 952 pts/1    S+   13:32   0:00 ./fill_mem.exe
13 1000     12362  0.0  0.0   4400   612 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
14 1000     12364  0.0  0.0   9384   920 pts/1    S+   13:32   0:00 grep fill_mem.exe
15 *************
16 allocation 3 ... n = 1230
17 1000     12350  0.0  0.0 14550804 956 pts/1    S+   13:32   0:00 ./fill_mem.exe
18 1000     12367  0.0  0.0   4400   612 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
19 1000     12369  0.0  0.0   9384   920 pts/1    S+   13:32   0:00 grep fill_mem.exe
20 *************
21 allocation 4 ... n = 1240
22 1000     12350  0.0  0.0 14908284 956 pts/1    S+   13:32   0:00 ./fill_mem.exe
23 1000     12372  0.0  0.0   4400   612 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
24 1000     12374  0.0  0.0   9384   920 pts/1    S+   13:32   0:00 grep fill_mem.exe
25 *************
26 allocation 5 ... n = 1250
27 1000     12350  0.0  0.0 15271572 956 pts/1    S+   13:32   0:00 ./fill_mem.exe
28 1000     12377  0.0  0.0   4400   612 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
29 1000     12379  0.0  0.0   9384   916 pts/1    S+   13:32   0:00 grep fill_mem.exe
30 *************
31 allocation 6 ... n = 1260
32 1000     12350  0.0  0.0 15640720 956 pts/1    S+   13:32   0:00 ./fill_mem.exe
33 1000     12382  0.0  0.0   4400   616 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
34 1000     12384  0.0  0.0   9384   920 pts/1    S+   13:32   0:00 grep fill_mem.exe
35 *************
36 allocation 7 ... n = 1270
37 1000     12350  0.0  0.0 16015776 956 pts/1    S+   13:32   0:00 ./fill_mem.exe
38 1000     12387  0.0  0.0   4400   616 pts/1    S+   13:32   0:00 sh -c ps aux | grep fill_mem.exe >> outfile
39 1000     12389  0.0  0.0   9384   920 pts/1    S+   13:32   0:00 grep fill_mem.exe

Now, I see that the VSZ portion gets up to ~15 GB so I am assuming when I try to address more, it fails with

Operating system error: Cannot allocate memory
Allocation would exceed memory limit

because there is not that much RAM. Why is it that RSS is so far below that, though? When I actually look on my system resources I see about 140 MB being used up (I am running this in a Linux VM and monitoring the system resources through Windows - I have given the GM 16 GB of RAM to use though, so I should see the VM memory increasing until it reaches the 16 GB mark - for what it's worth, the VM has VT-x/Nested Paging/PAE/NX so it should use the physical architecture just like the native OS).

Can anyone explain why I do not see my program actually using up the full 16 GB of RAM and how I can write my code to keep these arrays I am creating in RAM - fully utilizing my available hardware?

NOTE: The reason I am trying to write a sample program which reads a lot of memory is that I am working with data which takes up around 14 GB of space in ascii text. I will need to be working with data A LOT throughout the course of this program, so I want to read it all in at once and then reference it from RAM throughout the duration of the program. To make sure I am doing this correctly, I am trying to write a simple program which will store a very large array (~15 GB) in memory all at once.

Answer 1

(Caveat: The Fortran standard doesn't say how such thing ought to be implemented etc., the description below refers to how Fortran compilers are typically implemented on current operating systems.)

When you execute an ALLOCATE statement (or equivalently, calling malloc() in C, FWIW), you're not actually reserving physical memory, but only mapping address space for your process. That's why the VSZ goes up, but not the RSS. Actually reserving physical memory for your process happens only when you first access the memory (typically at page size granularity, that is, 4 KB on most current hw). So only once you start putting some data into your array does the RSS begin to climb. E.g. a statement like

real_arr = 42.

ought to bump up your RSS to the vicinity of the VSZ.

Answer 2

You probably need to increase the memory allocated to the stack. For example, see http://software.intel.com/en-us/articles/intel-fortran-compiler-increased-stack-usage-of-80-or-higher-compilers-causes-segmentation-fault