CODEWITHSUNDEEP
pentahopentaho-spoonpentaho-data-integration
user882670
user882670

Reputation:

Sort rows taking too long

I'm new to Pentaho.

The following transformation seems to be correct.

However, it is taking eternities to sort the rows:

enter image description here

This is the step:

enter image description here

Why is this taking so long and how can I improve this?

This is Merge join 2: e

Upvotes: 1

Views: 2553

Answers (3)

DisKanis
DisKanis

Reputation: 11

Step 'Merge join2' has Time-step as input twice. Because 'Sort rows' is not defined as input-step in 'Merge join2' the records from 'Sort rows' are not processed. If de buffer is on 10000 'Sort rows' stops writing to 'Merge join2'. Step 'Sort rows' will therefore never finish.

Upvotes: 1

NK0709
NK0709

Reputation: 66

few ways I tried to work with sort.. hope this helps

  1. Use the Sort size (rows in memory) setting on the Sort rows step to control this. The Free memory threshold (in %) helps avoid filling up available memory. Be sure to allocate enough RAM to PDI.
  2. Leverage database push the data to staging and sort and populate the target table works quick in term of batch processing.

Upvotes: 0

rmorales
rmorales

Reputation: 61

Try this in the transformation properties ( ctrl + T or View -> Settings ):

  • Set a bigger nr of rows in rowset
  • Set a bigger Feedback size
  • Uncheck "Manage Thread priorities?"

Example: Transformation properties

Upvotes: 2

Related Questions