Increase file size for Mex on Matlab

Question

I'm writing a FORTRAN mex file in Matlab r2019a. I've tested the code with small arrays and everything works fine. However, when I try and increase the size of the arrays I get the following error.

fatal error LNK1248: image size (9B993000) exceeds maximum allowable size(80000000)

I'm using Visual Studio 2017 and intel FORTRAN compiler on windows 10 and I've got 16GB of RAM. This link seems to have the solution, but I can't seem to find the linker option in VS 2017 to pass -heap-array.

https://software.intel.com/content/www/us/en/develop/articles/intel-fortran-compiler-increased-stack-usage-of-80-or-higher-compilers-causes-segmentation-fault.html

Could someone please walk me through how to increase the size?

Thank you Kinan

    #include "fintrf.h"
C     Gateway routine
      subroutine mexFunction(nlhs, plhs, nrhs, prhs)
       
       

C     Declarations
      implicit none



C     mexFunction arguments:
      mwPointer plhs(*), prhs(*)
      mwSize  nlhs, nrhs


C     Function declarations:
      mwPointer mxCreateDoubleMatrix
      mwPointer mxDuplicateArray

      mwPointer mxGetPr

      mwPointer mxGetM, mxGetN
      mwSize  mxIsNumeric 


C     Pointers to input/output mxArrays:
      mwPointer coord_pr,dualnumfam_pr,dualfail_pr,dualpointfam_pr
      mwPointer dualnodefam_pr,totnode_pr,width_pr,scr0_pr,vol_pr,bc_pr
      mwPointer disp_pr,numfam_pr,nodefam_pr,pointfam_pr,fail_pr
      mwPointer dmgpar1_pr,dmgpar2_pr, pforce_pr, dualpforce_pr
      mwPointer fails_pr,dualfails_pr


C     Array information:
      mwPointer m, n, x, y, u, l,o, p
      mwSize  mm, nn
      mwSize size, row, sizes
      
C     Arguments for computational routine:
      real*8  coord(450000), dualnumfam(300000) 
      real*8 dualfail(30000000), dualpointfam(150000) 
      real*8  dualnodefam(30000000),totnode(1) ,width(1)
      real*8 scr0(150000), vol(150000), bc(150000)
      real*8  disp(450000), numfam(150000),nodefam(30000000) 
      real*8  pointfam(150000),fail(30000000) 
      real*8  dmgpar1(150000),dmgpar2(150000)
      real*8  pforce(450000),dualpforce(450000)
      real*8  fails(30000000),dualfails(30000000)

        
      character*200 msg
      character*20  fmt
      character*10  sm, sn, sx, sy, so, sp
C-----------------------------------------------------------------------
    

C     Check for proper number of arguments. 
      if (nrhs .ne. 15) then
         call mexErrMsgIdAndTxt ('MATLAB:test:nInput',
     +                           'One inputs required.')
      endif

C     Validate inputs
C     Check to see both inputs are numeric.
      if (mxIsNumeric(prhs(1)) .ne. 1) then
         call mexErrMsgIdAndTxt ('MATLAB:test:NonNumeric1',
     +                           'Input # 1 is not a numeric.')
      endif







C     Check that input #1 is a scalar.
      m = mxGetM(prhs(1))
      n = mxGetN(prhs(1))
      size = m*n
      x = mxGetM(prhs(3))
      y = mxGetN(prhs(3))
        sizes=x*y
      u = mxGetM(prhs(5))


      fmt = '(I8)'
      write (sm,fmt) m
      write (sn,fmt) n
      write (sx,fmt) x
      write (sy,fmt) y
       
      msg = 'm=' // trim(sm) // ',\t n=' // trim(sn) // '\n'
      call mexPrintf(trim(msg))
      msg = 'x=' // trim(sx) // ',\t y=' // trim(sy) // '\n'
      call mexPrintf(trim(msg))

C     Create matrix for the return argument.
               call mexPrintf("one")

      plhs(1) = mxCreateDoubleMatrix(m,1,0)
      plhs(2) = mxCreateDoubleMatrix(m,1,0)
      plhs(3) = mxCreateDoubleMatrix(m,3,0)
      plhs(4) = mxCreateDoubleMatrix(m,3,0)
      plhs(5) = mxCreateDoubleMatrix(x,y,0)
      plhs(6) = mxCreateDoubleMatrix(x,y,0)
               call mexPrintf("two")


      coord_pr = mxGetPr(prhs(1))
      dualnumfam_pr = mxGetPr(prhs(2))
      dualfail_pr = mxGetPr(prhs(3))
      dualpointfam_pr = mxGetPr(prhs(4))
      dualnodefam_pr = mxGetPr(prhs(5))
      totnode_pr = mxGetPr(prhs(6))
      width_pr = mxGetPr(prhs(7))
      scr0_pr = mxGetPr(prhs(8))
      vol_pr = mxGetPr(prhs(9))
      bc_pr = mxGetPr(prhs(10))
      disp_pr = mxGetPr(prhs(11))
      numfam_pr = mxGetPr(prhs(12))
      nodefam_pr = mxGetPr(prhs(13))
      pointfam_pr = mxGetPr(prhs(14))
      fail_pr = mxGetPr(prhs(15))
      dmgpar1_pr = mxGetPr(plhs(1))
      dmgpar2_pr = mxGetPr(plhs(2))
      pforce_pr = mxGetPr(plhs(3))
      dualpforce_pr = mxGetPr(plhs(4))
      fails_pr = mxGetPr(plhs(5))
      dualfails_pr = mxGetPr(plhs(6))


               call mexPrintf("three")


C     Load the data into Fortran arrays.
      call mxCopyPtrToReal8(coord_pr,coord,size)    
      call mxCopyPtrToReal8(dualnumfam_pr,dualnumfam,m)
      call mxCopyPtrToReal8(dualfail_pr,dualfail,sizes)
      call mxCopyPtrToReal8(dualpointfam_pr,dualpointfam,m)
      call mxCopyPtrToReal8(dualnodefam_pr,dualnodefam,u)
      call mxCopyPtrToReal8(totnode_pr,totnode,1)
      call mxCopyPtrToReal8(width_pr,width,1)
      call mxCopyPtrToReal8(scr0_pr,scr0,m)
      call mxCopyPtrToReal8(vol_pr,vol,m)
      call mxCopyPtrToReal8(bc_pr,bc,m)
      call mxCopyPtrToReal8(disp_pr,disp,size)
      call mxCopyPtrToReal8(numfam_pr,numfam,m)
      call mxCopyPtrToReal8(nodefam_pr,nodefam,u)
      call mxCopyPtrToReal8(pointfam_pr,pointfam,m)
      call mxCopyPtrToReal8(fail_pr,fail,sizes)
      call mxCopyPtrToReal8(dmgpar1_pr,dmgpar1,m)
      call mxCopyPtrToReal8(dmgpar2_pr,dmgpar2,m)
      call mxCopyPtrToReal8(pforce_pr,pforce,size)
      call mxCopyPtrToReal8(dualpforce_pr,dualpforce,size)
               call mexPrintf("four")

      call mxCopyPtrToReal8(fails_pr,fails,sizes)
      call mxCopyPtrToReal8(dualfails_pr,dualfails,sizes)



C     Call the computational subroutine.
        
      call body(coord,dualnumfam,dualfail,dualpointfam,dualnodefam,
     + totnode,width,scr0,vol,bc,disp,numfam,nodefam,pointfam,fail
     + ,m,n,dmgpar1,dmgpar2,pforce,dualpforce,u,fails,dualfails)
C     Load the output into a MATLAB array.
      call mxCopyReal8ToPtr(dmgpar1,dmgpar1_pr,m)
      call mxCopyReal8ToPtr(dmgpar2,dmgpar2_pr,m)
      call mxCopyReal8ToPtr(pforce,pforce_pr,size)
      call mxCopyReal8ToPtr(dualpforce,dualpforce_pr,size)
      call mxCopyReal8ToPtr(fails,fails_pr,sizes)
      call mxCopyReal8ToPtr(dualfails,dualfails_pr,sizes)


      return
      end

Answer 1

The stack is a relatively small amount of memory compared to the heap. The stack is a fixed amount of memory that is essentially part of your program. It is used for such things as passing arguments among routines, local variable memory, etc. Because of its relatively small size, you should not create large local variables that can overflow the stack. This is true in any language, not just Fortran. So creating a local variable like this:

real*8 dualfail(30000000)

causes dualfail memory to come from the stack.

A better method for large variables is to allocate memory for them from the heap, which is essentially your entire main computer memory. E.g.,

real*8, allocatable :: dualfail(:)
integer alloc_stat
allocate(dualfail(30000000),stat=alloc_stat)
if( alloc_stat /= 0 ) then
    ! allocation failed, so take action here
endif
! code to use dualfail here
deallocate(dualfail)

All of your large variables should use this technique.

Having said that, it appears that the only reason you have these large variables in your mex routine is to make copies of the MATLAB inputs & outputs for the body( ) routine. This is a very inefficient way of managing this. You are also deep copying your output variables which are just 0's even though I am guessing they get overwritten by body( ) anyway. Rather than taking this approach, especially when working with large variables, it would be best to simply pass "pointers" to the memory to your routines. One way to do this is using the %VAL( ) construct. This will eliminate the need to create these large local variables in the first place. E.g.,

  call body(%VAL(coord_pr),%VAL(dualnumfam_pr),%VAL(dualfail_pr), etc.

So everywhere you were passing copies of the MATLAB variable data, you would instead pass the "pointers" (actually passing the address contained in the integer by value) to the original data areas of the MATLAB variables. As long as your body( ) routine treats the MATLAB prhs( ) inputs as read-only then this will work without the need to make deep copies as you are doing. This eliminates the need for all of those mxCopyReal8ToPtr( ) and mxCopyPtrToReal8( ) calls.

Note that my use of the word "pointers" above is used generically ... these are not Fortran pointer variables. But using actual Fortran pointers would be another method for avoiding deep data copies ... i.e., turn the integer returned by mxGetPr( ) into a regular Fortran pointer and then use that downstream in your code.

I would also point out that your mex routine is severely lacking in input argument checks. You only check for the number of inputs and that the first input is numeric. What you should be doing is checking that each and every one of your inputs is double, non-complex, non-sparse, and is of the correct size. As it is the routine is not robust and you are at risk of getting erroneous results or crashing MATLAB if the inputs are not exactly as expected.

Finally, be advised that for some reason The Mathworks has chosen to hard code the /fixed option into their Fortran mex build files. I have asked them to remove this but as of R2020a it is still there. This forces the compiler to treat all files as fixed format, even those ending in .f90. My advice is to find those xml build files on your system and remove the /fixed option from those files. That way you can naturally compile .f90 files as free format.