CODEWITHSUNDEEP
javaout-of-memorypentaho
Lucas Rezende
Lucas Rezende

Reputation: 2686

Pentaho OutOfMemory Exception with a 12GB Ram Machine

I just implemented Pentaho in my company and set its memory with 12GB. When we try to load a 16 million rows from one table to another, it goes out of memory.

I thought Pentaho would clear the memory when performing the commit on the database, but seemingly it doesn't happen. This exception is thrown when it loads around row 2.5 million, which means to load 16 million I'd need a 73Gb RAM machine? (a rough math, of course)

Is there any parameter or configuration to make the magic happen? This memory issue is limiting our loading capacity (16 million is only one of the tables ). Can't believe Pentaho will braise the memory until it bursts without clearing the cache eventually.

My file D:\Pentaho\server\biserver-ee\tomcat\bin\service.bat has the following line:

"%EXECUTABLE%" //US//%SERVICE_NAME% ++JvmOptions "-Djava.io.tmpdir=%CATALINA_BASE%\temp;
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager;
-Djava.util.logging.config.file=%CATALINA_BASE%\conf\logging.properties;
-XX:MaxPermSize=256m" --JvmMs 2048 --JvmMx 12288

Does it have anything to do with the line below?

-XX:MaxPermSize=256m

Could someone explain me what exactly it is?

Thanks in advance!

PS: This is my first contact with Pentaho, so, I am sorry for any unnecessary question or assumption.

Upvotes: 6

Views: 6110

Answers (2)

lukfi
lukfi

Reputation: 319

Some ETL steps have to read (and therefore cache) all the data before they start giving results (e.g. Memory Group by, Stream lookup for the lookup stream). But if you only read (table input) and write (table output), data just goes in and out and you don't need to fit the whole table into memory (that would be pretty useless, right?).

The --JvmMs 2048 --JvmMx 12288 parameters look suspicious to me. Have you tried -Xms2g -Xmx12g ?

Upvotes: 2

SteveD
SteveD

Reputation: 5405

On the MaxPermSize switch, Oracle Java versions prior to 8 have an area of memory called permgen (permanent generation).

See this answer for more detail on it.

This can be a source of out-of-memory exceptions, though not knowing Pentaho and your usage, it's hard to say if that's the source of your problem.

Upvotes: 3

Related Questions