CODEWITHSUNDEEP
apache-nifi
Krakenudo
Krakenudo

Reputation: 309

NiFi How can I merge exactly 3 records in a single one?

I'm working with NiFi, receiving several json in the same file. Pretending to modify propertly those jsons, I split it into several single flowfiles and here is where the problems start.

For each id and datetime, there are 3 flowfiles that I would like to merge into a single flowfile. I've tried to use MergeRecord NiFi's Processor, but it works when it wants. When I have a few records, it seems that it works ok. But when I have "a lot of" records, e.g., more than 70 records, it breaks. Sometimes it merges 2 records in a single record, sometimes it lets pass a single record directly.

NiFi's MergeRecord processor

merge_key is a string attribute based on id and datetime.

I need to take exactly 3 records and merge them in a single one.

If there is a way to order the flowfile and take the first n elements of it each 5 seconds, I think it can help. But I prefer to be sure it works properly without any "help"...

Upvotes: 0

Views: 2377

Answers (1)

Manoj
Manoj

Reputation: 390

For ordering the flowfile we can use the EnforceOrder processor as per the default documentation.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.EnforceOrder/#:~:text=Order%20Attribute,will%20be%20routed%20to%20failure.

Also NiFi uses below prioritizers among the incoming flow files.

  • FirstInFirstOutPrioritizer

  • NewestFlowFileFirstPrioritizer

  • OldestFlowFileFirstPrioritizer

  • PriorityAttributePrioritizer

Refer below link for more details

https://nifi.apache.org/docs/nifi-docs/html/user-guide.html

Upvotes: 2

Related Questions