CODEWITHSUNDEEP
apache-nifi
ashok
ashok

Reputation: 1268

NiFi after calling HTTP API, how to get original flowfile before calling HTTP API

I have the sequence below for calling the Rest API.

GenerateFlowFile
  ↓
EvaluateJSONPath
  ↓
ReplaceText (for post data creation)
  ↓
InvokeHTTP
  ↓
EvaluateXPath
  ↓
Original FlowFile (which was generated by GenerateFlowFile)

So, after the 'ReplaceText' processor the original data will be replaced with new data.

How can I get the original data and use the attributes produced after calling API?

Upvotes: 7

Views: 2345

Answers (3)

Dennis Jaheruddin
Dennis Jaheruddin

Reputation: 21563

The two generic answers are:

  1. MergeContent as explained by dagget
  2. Wait/Notify as described here (originally by Andy)

These solutions should scale well and be considered in the first place. I personally find them a bit complex, so therefore I also propose a worarkound here.

If you are doing HTTP requests, perhaps you may only be processing a small amount of very small messages. In this case you can consider the following 'trick' to avoid complexity.

Rather than splitting, consider (ab)using an attribute

The easiest fix is by keeping all data together. Rather than splitting your message, just put a copy of the original content in an attribute. This attribute will still be available after your HTTP request, regardless of what is returned in the content.

The simple solution would be to use an ExtractText right before the HTTP request, and create an attribute called something like original with the whole content.

Mandatory warning: Attributes are designed to be small and are therefore stored in memory. Hence putting large content in an attribute may consume your memory quickly.

A final workaround for possible future readers: If you control the HTTP service, or at least know the specs, consider whether it is desirable to let the output contain the input as well. Often not, but sometimes you get it anyway!

Upvotes: 0

daggett
daggett

Reputation: 28564

at the point where you have original file insert UpdateAttribute with evaluation of some unique attribute.

for example MyUID = ${UUID()}

the success connection after UpdateAttribute should go to preparation flow to invoke http, and the copy of this connection should go to MergeContent that should combine original and evaluated content & attributes.

flow: enter image description here

UUID & split: enter image description here

merge content: enter image description here

Upvotes: 2

Andy
Andy

Reputation: 14184

You can either maintain the original flowfile with a direct output relationship from GenerateFlowFile and merge the flowfiles later with MergeContent with mode Defragment and Keep All Unique Attributes, or if the original flowfile content is small enough, you can move it to an attribute before changing the flowfile content and then recombine them after you receive the new data with Update Attribute/ReplaceText.

Upvotes: 1

Related Questions