CODEWITHSUNDEEP
jsonmergeapache-nifidataflow
3nomis
3nomis

Reputation: 1613

How to merge partitioned Json in Apache Nifi

Hi I am splitting Json files of varying size from 2 GB to 3 GB.
How should I set the 'Merge content' processor such as to have the original file?
I am facing the issue of having multiple part of the same flow file after doing the merge, thi might be I am setting the limits wrong.
Do you know hot to solve this issue?
I am focusing on these properties:

-Minimum Number of Entries
-Maximum Number of Entries
-Minimum Group Size
-Maximum Group Size

Upvotes: 0

Views: 1165

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18670

To reconstruct a split file you need to use MergeContent in defragment mode. I believe this ignores all of the entry and size properties and just uses the "fragment" attributes that were placed on each flow file by the split processor, and merges all the fragments back together.

Upvotes: 2

Related Questions