CODEWITHSUNDEEP
talend
DrGenius
DrGenius

Reputation: 967

How to fix Java heap space error in Talend?

I have an ETL flow through talend and there:

  1. Read the zipped files from a remote server with a job.
  2. Take this files unzipes them and parse them into HDFS with a job. Inside the job exists a schema check so if something is not

My problem is that TAC server stopes the execution because of this error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at org.talend.fileprocess.TOSDelimitedReader$ColumnBuffer4Joiner.saveCharInJoiner(TOSDelimitedReader.java:503) at org.talend.fileprocess.TOSDelimitedReader.joinAndRead(TOSDelimitedReader.java:261) at org.talend.fileprocess.TOSDelimitedReader.readRecord_SplitField(TOSDelimitedReader.java:148) at org.talend.fileprocess.TOSDelimitedReader.readRecord(TOSDelimitedReader.java:125) ....

Is there any option to avoid and handle this error automatically? There are only few files which cause this error but I want to find a solution for further similar situation.

Upvotes: 1

Views: 12514

Answers (3)

Budi Odank
Budi Odank

Reputation: 11

job design talend

Success add arguments -Xmx6144M and -Xms2048M for migrate (insert) 3 million data transfer database postgree in Talend Open Studio version 8.0.1

Upvotes: 0

VimalK
VimalK

Reputation: 75

@DrGenius Talend has java based environment and some default jvm heap is awarded during initialization, as in for any java program. Default for Talend - Min:256MB (xms) & max:1024MB.. As per your job requirement, you can set the range of min/max jvm like min of 512 mb & max 8gb..

This can be modified from Job run tab - advance setting.. Even this can be parameterized and can be overwritten using variables set in env. Exact value can be seen from job build -> _run.sh .. But be careful not to set high as too high so that other jobs running on same server is depleted of memory.

More details on heap error & how to debug issue: https://dzone.com/articles/java-out-of-memory-heap-analysis

Upvotes: 1

RealHowTo
RealHowTo

Reputation: 35407

In the TAC Job Conductor, for a selected job, you can add JVM parameters.

enter image description here

Add the -Xmx parameter to specify the maximum heap size. The default value depends on various factors like the JVM release/vendor, the actual memory of the machine, etc... In your situation, the java.lang.OutOfMemoryError: Java heap space reveals that the default value is not enough for this job so you need to override it.

For example, specify -Xmx2048m for 2048Mb or 2gb

Upvotes: 3

Related Questions