CODEWITHSUNDEEP
apache-nifi
natarajan k
natarajan k

Reputation: 406

NiFi: how to store maxTimestamp when using ListFile/GetFile processor?

i am using MiNiFi 0.3 and NiFi 1.5 version.

we have a requirement to pull the data(csv) from 'A' folder using MiNiFi and send to NiFi running in linux.

for instance, if file is arriving with 10 records on 1 am. we need to move(not copy) file from 'A' folder to NiFi hub. After 10 minutes (1.10 am), the appended file will be arriving with the older 10 records and new 10 records. so, totally it will contain 20 records.

we need to send only the new 10 records to the NiFi hub.

i tried ListFile -> FetchFile, but since we need to move the data. this does not work. then i tried with GetFile processor, but it captures the whole 20 records.

is there any way to achieve the scenario.

thanks in advance.

Upvotes: 0

Views: 271

Answers (1)

Ben Yaakobi
Ben Yaakobi

Reputation: 1658

Using FetchFile, you can configure it using property Completion Strategy to Move File or even Delete File(and then you can PutFile it whenever you like).

Upvotes: 1

Related Questions