CODEWITHSUNDEEP
jsonbigdataapache-nifidata-processing
3nomis
3nomis

Reputation: 1613

Merge Json after several splits in Nifi

I splitted my Json several times to avoid OOM errors. I've put a Wait processor to wait for all my records to the use a Merge content. Each FF has been assigned an attribute of the original file number of lines.

The wait processor should put the FF in wait until the notify increases the counter to the total number of lines.

However It seems that my Wait processor is not putting my FF in the Wait queue(it is not shown but there is).

Is there anything wrong in this peace of flow?

enter image description here

Upvotes: 1

Views: 894

Answers (1)

mattyb
mattyb

Reputation: 12093

You can do multiple merges by using UpdateAttribute after each Split to save the fragment.* attributes as something different, perhaps fragment1.*, fragment2.*, etc. Then you can restore each of them in reverse order with UpdateAttribute before each Merge, setting fragment.* to the fragment2.* attributes, then MergeContent, then set fragment.* to the fragment1.* attributes, then MergeContent, and so on.

Upvotes: 1

Related Questions