CODEWITHSUNDEEP
rmemory
Cyang
Cyang

Reputation: 379

R: Cannot allocate memory greater than x MB

I have a main function in R which calls other files to run my program. I call the main file through a bat file(.exe). When I run it line-by-line it runs without a memory error, but when I call the bat file to run it, it halts and gives me the following error:

Cannot allocate memory greater than 51 MB.

How can I avoid this?

Upvotes: 3

Views: 1190

Answers (2)

Hack-R
Hack-R

Reputation: 23200

Memory limitations in R such as this are a recurring nightmare for a lot of us.

Very often the problem is a limit imposed by your OS limits (which can usually be changed on a Bash or PowerShell command line), architecture (32 v. 64 bit), or the availability of contiguous free RAM, irregardless of overall available memory.

It's hard to say why something would not cause a memory issue when run line by line, but would hit the memory limit when run as a .bat.

What version of R are you running? Do you have both installed? Is 32-bit being called by Rscript when you run your .bat file whereas you run a 64-bit version line by line? You can check the version of R that's being run with R.Version().

You can test this by running the command memory.limit() in both your R IDE/terminal and in your .bat file (be sure to print or save the result as an object in your .bat file). You might also do well to try setting memory.limit() in your .bat file, as it may just have a smaller default, perhaps due to differences in your R Profile that's invoked in your IDE or terminal versus the .bat file.

If architecture isn't the cause of your memory error, then you have several more troubleshooting steps to try:

  1. Check memory usage in both environments (in R directly and via your .bat process) using this: sort( sapply(ls(),function(x){object.size(get(x))}))
  2. Run the garbage collector explicitly in your scripts, that's the gc() command
  3. Check all object sizes to make sure there are no unexpected results in your .bat process: sort( sapply(ls(),function(x){format(object.size(get(x)), units = "Mb")}))

  4. Try memory profiling: 

    Rprof(tf <- "rprof.log", memory.profiling=TRUE)
Rprof(NULL)
summaryRprof(tf)

  5. While this is a RAM issue, for good measure you might want to check that the compute power available is both sufficient and not varying between these two ways of running your code: parallel::detectCores()

  6. Examine your performance with Prof. Hadley Wikham's lineprof tool (warning: requires devtools and doesn't work on lines of code which call the C programming language)

    References While I'm pulling these snippets out of my own code, most of them originally came from other, related StackOverflow posts, such as:

    1. Reaching memory allocation in R
    2. R Memory Allocation "Error: cannot allocate vector of size 75.1 Mb"
    3. R memory limit warning vs "unable to allocate..."
    4. How to compute the size of the allocated memory for a general type
    5. R : Any other solution to "cannot allocate vector size n mb" in R?

Upvotes: 2

Rusan Kax
Rusan Kax

Reputation: 1894

Yes you should be using 64bit R, if you can.

See this question, and this from the R docs.

Upvotes: 1

Related Questions