How to solve Pentaho text file output error - Requested array size exceeds VM limit

Question

I am exporting data by suing "table input" step to "text_file_output" step into csv file, but failing to export even 10 records in the csv file because of "java.lang.OutOfMemoryError: Requested array size exceeds VM limit" My spoon.bat file has below parameters already set but no luck yet. PENTAHO_DI_JAVA_OPTIONS="-Xms1024m" "-Xmx2048m" "-XX:MaxPermSize=256m" "-XX:+UseCompressedOops" "-XX:+HeapDumpOnOutOfMemoryError"

In this transformation, exporting data to excel works okay but performance is bad for 10k+ records , so used alternate way as text file output with csv extension, tested with sample data and it worked for 10k records in few seconds only but for new data it's failing even for 1 record. Tried with increasing heap size as Xms 2GB and Xmx 4 GB but no luck

Answer 1

I've been struggling for an hour on this problem and it finally was only about a stupid default length Pentaho set on some String fields in the "fields" tab of the Text File Ouput step.

Just set the string's lenght to your needs, which should be way below the default one, or set it empty, and everything shall work like a charm, including formatting.

NB In my case Pentaho set this huge default length to variable lenght postgresql fields (varchar)

Answer 2

Try ticking the "Fast data dump (no formatting)" box in the Content section of Text file output step.