CODEWITHSUNDEEP
schedulerapache-nifi
Geeta Singh
Geeta Singh

Reputation: 181

Scheduling job with Apache NiFi by passing dynamic property values

I have created a NiFi workflow as shown below: GenerateFlowFile --> Custom Processor --> LogAttribute My custom processor has a property as start date. But the start date should change in each scheduled run based on the maximum end date from previous run. Basically looking for incremental data fetch from the server.

Could you please help, how this can be achieved in Apache NiFi?

Upvotes: 1

Views: 2290

Answers (1)

James
James

Reputation: 11931

Processor scheduling is usually left to the data flow manager configuring the processor into their flow. I recommend you let them schedule the processor, expecting it to run on a periodic basis.

But you can use Apache NiFi's State Manager feature to store data that tracks your incremental progress. You could then decide what action to take, if any, when the processor is triggered. If there is nothing to do, don't do anything.

The best examples of this are List* processors like ListFile. These processors typically store a timestamp of the file they last read, the use that timestamp to determine which newer files should be acted on, regardless of how frequently they are asked to check. It is likely that most executions of a List* processor will result in no output.

There are some examples of reading and persisting state data in the AbstractListProcessor class.

Upvotes: 3

Related Questions