CODEWITHSUNDEEP
apache-nifikylo
Rakesh Prasad
Rakesh Prasad

Reputation: 642

running record count from SplitRecord processor Nifi

Is there a way to get fragment index from SplitRecord processor Nifi? I am splitting a very big xls (4 mill records) into "Records Per Split" = 100000.

Now I want to just process first 2 splits, to see quality of the file and reject rest of the file.

I can see fragment index is in other split function (e.g. JsonSplit), but not in record split. Any other hack?

Upvotes: 1

Views: 1365

Answers (1)

notNull
notNull

Reputation: 31520

Method1:

By using Control Rate processor we can achieve this case

Control Rate Processor: enter image description here

By this configs we are releasing 2 flowfiles for every minute and

Flow: enter image description here

Configure the queue expiration to like 10 sec(or lower number if you need), then the flowfiles are going to expired in the queue but first 2 flowfiles are going to be released.

Method2:

By using SplitText processor then use RouteOnAttribute Processor and add new property as

${fragment.index:le(2)}

By using above expression language we are only allowing only the first 2 fragment indexes.

Refer to this link for splitting Big File in NiFi.

Upvotes: 3

Related Questions