How to merge streams in Pentaho?

Question

I've got a job that loads three Excel-files from FTP to local drive and then launch three different transformations doing som normalization etc. and then output the data again to Excel and finally put them to FTP a site again:

This is the main job:

The main job starts three of this, each converting from one Excel file to another. Every output Excel file is just rows with data with equal number of columns. WHat I'd like to do is to merge the rows into one file.

How can this be done?

Answer 1

You can also try out the 'Append streams' step. Again, all of the Excel inputs would have to have matching metadata for this to work.

Answer 2

All steps in Pentaho can take multiple inputs as long as stream metadata matches exactly. So, make sure that all three Excel inputs have exactly matching metadata, meaning field names, data types, and field order must be exactly the same, then just flow the outputs into the same step like this:

Note, the columns will not be ordered. This is like the Union All step in SSIS. If you have sorted inputs and wish the output to be sorted also, use a Sorted Merge step.