CODEWITHSUNDEEP
apache-nifi
natarajan k
natarajan k

Reputation: 406

nifi: how to merge multiple columns in csv file?

nifi version: 1.5

input file:

col1,col2,col3,col4,col5,col6
a,hr,nat,REF,6,2481
a,hr,nat,TDB,6,1845
b,IT,raj,NAV,6,2678

i want to merge the last three columns with : delimiter and separator by / based on col1.

expected output:

col1,col2,col3,col4
a,hr,nat,REF:6:2481/TDB:6:1845
b,IT,raj,NAV:6:2678

i am not able to find the solution because lot of response were based merging two files. is there a better way to do it?

tia.

Upvotes: 0

Views: 1127

Answers (1)

mattyb
mattyb

Reputation: 12083

I think you'll want a PartitionRecord processor first, with partition field col1, this will split the flow file into multiple flow files where each distinct value of col1 will be in its own flow file. If the first 3 columns are to be used for partitioning, you can add all three columns as user-defined properties for partitioning.

Then whether you use a scripted solution or perhaps QueryRecord (if Calcite supports "group by" concatenation), the memory usage should be less as you are only dealing with a flow file at a time whose rows are already associated by the specified group.

Upvotes: 1

Related Questions