CODEWITHSUNDEEP
apache-nifi
Dan
Dan

Reputation: 4329

Method for collecting user input within NiFi topology

I'm new to NiFi, so I'm not sure if this is possible (or a correct design approach):

I'm trying to create a processing pipeline where NiFi fetches a file (potentially multiple gigabytes in size) and performs some processing -- pretty straightforward until ...

A small portion of the resultant data needs to be shown to an end-user. The user would provide an input, and then NiFi will need to perform follow-on processing of the original data set.

I was going to use a Wait processor, prompt the user (via a GUI) using a PostHTTP processor, and then use a Notify processor. The flow looks like this:

FetchFile ==> Initial Processing ==> Wait

Notify ==> Follow-on Processing

Unfortunately, the PostHTTP processor posts ALL of the FlowFile's content to the specified endpoint ... which, again, could be multiple gigabytes in size ... which is prohibitive.

Is there a "standalone" NiFi processor (which doesn't include the FlowFile content) for this sort of User interaction? Is there another implementation strategy for this use-case? Or is this not even a correct application of NiFi?

Thank you!

Upvotes: 2

Views: 581

Answers (1)

James
James

Reputation: 11931

A possible approach would be to write the file out to a temporary store (local disk, HDFS, S3, etc.), write a request for user action, and essentially exit NiFi completely. The flow would then be restarted after the user has taken action. To trigger the restart, you could have the UI post to NiFi, write to a message queue, or write another file to a directory NiFi is polling.

I think this would be more flexible for managing the user interaction, since you wouldn't have to complete the whole UI response in an HTTP post cycle.

Upvotes: 3

Related Questions