Reputation: 801

Java out of memory: increase heap space?

This seems to be a common issue, however the existing solutions didn't work for me. I am trying to perform topic modeling in R with the help of the mallet package. The corpus consists of forum comments and is app. 50 MB large. It is divided in 41.975 files.

Here's my script:

documents_65 <- mallet.read.dir("~/20170315_F65/tm")  

# Loading corpus and stopwords
mallet.instances <- mallet.import(id.array = documents_65$id, 
                                  text.array = documents_65$text,
                              "~/stopwords.txt", 
                              token.regexp = "\\p{L}[\\p{L}\\p{P}]+\\p{L}")
topic.model <- MalletLDA(num.topics = 1000)
topic.model$loadDocuments(mallet.instances)
vocabulary <- topic.model$getVocabulary()
word.freqs <- mallet.word.freqs(topic.model)
topic.model$setAlphaOptimization(10, 30)
topic.model$train(1000)
topic.model$maximize(10)
doc.topics <- mallet.doc.topics(topic.model, smoothed=T,
                            normalized=T)

After this I get the error:

Error in .jcall(wrapper, "[D", "flat_double") : 
java.lang.OutOfMemoryError: Java heap space

As many have suggested, I tried

options(java.parameters = "-Xmx1000m")

After that I tried the mallet.doc.topics command again and got the same error. I tried to assign the heap space in the Control Panel (as suggested here: https://www.youtube.com/watch?v=b-D24vnuUMM).

I also tried

gc()

Output:

           used  (Mb) gc trigger  (Mb) max used  (Mb)
Ncells  2407281 128.6    6619081 353.5  7415161 396.1
Vcells 32395876 247.2   48628929 371.1 48628929 371.1

as well as assigning more heap space through the command line with this:

java -Xmx2048m

I even tried to peak into the assigned memory with the XLConnect-package and the following commands:

xlcMemoryReport ()

Output:

Amount of free memory in the Java Virtual Machine (JVM):  308.0197 MB

xlcFreeMemory()


.jcall(.jnew("java/lang/Runtime"), "J", "totalMemory")
.jcall(.jnew("java/lang/Runtime"), "J", "maxMemory")

I hope I am not missing something really basic. I can imagine that my corpus is hard to handle and requires a lot of memory (mostly because of the large amount of files).

UPDATE: I couldn't resolve this issue in R and switched to mallet directly. There I found the batch file with specs for mallet, in which one could see that the java memory allotted to mallet was limited to 1 GB. Since my corpus was around 1.3 GB it wasn't possible at all to load it into Mallet. In the batch file I could easily change this and kept working with mallet directly.

I don't know, if it possible to access the batch file from R.

Upvotes: 2