CODEWITHSUNDEEP
apache-nifi
Mohammadreza Khedri
Mohammadreza Khedri

Reputation: 2711

Nifi merge content by similar attribute value

I have zip files like bellow in the directory:

139909-JSON.zip
139910-JSON.zip
...

And the content of each zip file is like this:

1
  672748
    1399101-672748-content.json
    1399101-672748-header.json
2
  684919
    1399101-684919-content.json
    1399101-684919-header.json
...

Im using the GetFile > UnpackContent > UpdateAttribute -> MergeContent flow right now. But I cant merge truly my data.

GetFile: get all zip files.

UnpackContent: unpack zip files and get all .json files.

UpdateAttribute: I have 672748 and 684919 from filename in the TracingNo variable.

I need to merge every file with the same TracingNo value together in one JSON file.

How can I do this?

Upvotes: 1

Views: 2237

Answers (2)

daggett
daggett

Reputation: 28634

Problem in this statement: Minimum 1 and maximum 2 files for each TracingNo.

It'll be better to know number of files to make merge content

However try to do it in 2 steps:

MergeContent with parameters:

Merge Strategy =    Bin-Packing Algorithm
Merge Format =  Binary Concatenation
Correlation Attribute Name = TracingNo
Minimum Number of Entries = 2
Maximum Number of Entries = 2
Max Bin Age = 10 seconds
Delimiter Strategy = Text
Header = [
Footer = ]
Demarcator = ,

This processor will wait for a second file during 10 sec. The single files will be classified as failed and you could use ReplaceText to add [] into them.

Upvotes: 1

Mike R
Mike R

Reputation: 589

What if you used a RouteAttribute to have all flowfiles with the same TracingNo routed the same way to have it route to the same MergeContent. This can be done to RouteAttribute so that each TracingNo get sent their own way, so that all flowfiles in each direction have the same TracingNo to allow for you to handle the same values and combine them

Upvotes: 1

Related Questions