CODEWITHSUNDEEP
apache-nifi
azelix
azelix

Reputation: 1277

Watch new files in directory NIFI

I'm having a use case where I have new files everyday at differents moments like every hour or two hours so I need to watch a directory in my folder, and on adding new files it triggers an event which sends those new files paths to my webservice on NIFI , any idea how to implement this and what tool to use for this ?

Or maybe this is not the best approach ?

Upvotes: 2

Views: 7533

Answers (2)

Bryan Bende
Bryan Bende

Reputation: 18630

Take a look at the ListFile and FetchFile processors:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ListFile/index.html

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.FetchFile/index.html

Complete NiFi documentation can be found at https://nifi.apache.org/docs.html

Upvotes: 10

khushbu kanojia
khushbu kanojia

Reputation: 250

If your file is in file sytem then use 'GETFILE' processor which on adding new file on provided 'input directory' triggers an event and immediately feed data into NIFI without any delay.

If your requirement is to schedule it like every hour or any specific time then use 'Scheduling' tab present on each processor's configuration and schedule it using 'Cron-Driven' strategy and set cron for every hour like this */60 * * * *?

If your file is in S3 bucket then you have to use SQS queue notification using 'GETSQS' processor documented in detailed in below link

http://crazyslate.com/apache-nifi-intergration-with-aws-s3/

https://community.hortonworks.com/content/idea/76919/how-to-integrate-aws-sqs-queue-with-nifi.html

Upvotes: 3

Related Questions