CODEWITHSUNDEEP
apacheapache-nifi
Rahul Varma
Rahul Varma

Reputation: 550

unable to get file linecount of files in nifi

I will be getting csv files (daily once around 10-12 csv files ) and using nifi we need to get the linecount of all csv files.

I have done this getfile -> counttext -> replacetext (in replace text i have replaced it with below value) enter image description here

now i am getting output as 12 files and each file has linecount of that files now how to add those numbers in nifi

sample output files file1:- 40 file2:- 35 file3:- 55 here i need to get one file where my value should be sum of all numbers in all files ex:- 130 (as per above data)

Upvotes: 1

Views: 580

Answers (1)

Pushkr
Pushkr

Reputation: 3619

You can use combination of getfile -> CountText -> ReplaceText -> mergeContent->QueryRecord to get sum of all counts as flowfile by running a simple sql query.

After your replaceText processor as mergeContent processor with two settings :

Minimum Number of Entries= <set this to minimum number of files you expect i.e. 10 or 12 Demarcator= Shift + Enter for newline character

enter image description here

Add QueryReecord processor with CSVREADER and CSVRECORDSETWRITE controller service.

enter image description here

You can use below avro schema as 'schema.text' in both controller service 

{
  "type": "record",
  "name": "test",
  "fields": [
    {
      "name": "cnt",
      "type": "long"
    }
  ]
}

Upvotes: 1

Related Questions