CODEWITHSUNDEEP
apache-nifi
Rakesh Prasad
Rakesh Prasad

Reputation: 642

ListFile processor, force processor to list full directory everytime

My use case.

Some processing somewhere else add files to some dir (_use_it) -> call my flow using REST -> Now I want my process to read all files from mentioned directory (_use_it).

I want to read all files everytime from this directory, not just changed/added files. I can't start/stop process. This flow has to run as a background process.

I think, I am looking for ListFile processor to run once, then stop, and then when It runs again, it forgets previous state. "some twisted logic" :)

Thanks

Upvotes: 3

Views: 2957

Answers (1)

notNull
notNull

Reputation: 31470

1. Using GetFile Processor:

You can use GetFile processor instead of ListFile + FetchFile processors and GetFile processor doesn't store the state.

  • GetFile processor Gets all the files in the directory every time.

  • Keep Source File property If true, the file is not deleted after it has been copied to the Content Repository; this causes the file to be picked up continually and is useful for testing purposes. If not keeping original NiFi will need write permissions on the directory it is pulling from otherwise it will ignore the file.

(or)

2. Using ListFile Processor:

Making use of NiFi RestAPI we can clear the state of list file processor and then processor will list out all files in the directory every time.

Clear state of the processor: 

POST 
/processors/{id}/state/clear-requests

Before you are starting the Listing all files in the directory flow

  1. Use Rest Api to stop the ListFile processor

  2. Clear the state of ListFile processor

  3. Start the ListFile processor.

Refer to this and this links to STOP the processor via RestApi

Upvotes: 6

Related Questions