CODEWITHSUNDEEP
apache-nifi
djgcp
djgcp

Reputation: 313

How to get NiFi processor error into Flowfile attribute?

I have a PutGCSObject processor for which I want to capture the error into a flow file attribute.

enter image description here

As in the Picture, when there is an error for the Processor, it sends to failure with all the pre-existing attributes as-is.

enter image description here

I want the error message to be a part of the same flow file as an attribute. How can I achieve that ?

Upvotes: 0

Views: 4252

Answers (2)

Up_One
Up_One

Reputation: 5271

There is actually a way to get it.

Here is how i do it:

1: I route all ERROR connections to a main "monitoring process group"

2: Here is my "monitoring process group"

enter image description here

In updateattribute I capture filename as initial_filename

Then in my next step I query the bulletins

enter image description here

I then parse the output as individual attributes.

After I have the parsed bulleting output I use a RouteOnAttribute proc to drop all bulletins I don't need (some of them I have already used and notified on).

enter image description here

Once I only have my actual ERROR bulletin left, I use ExecuteStreamingCommand to run a python script using nipyapi module to get more info about the error, such as where it is in my flow, hierarchy, a description of the processor that failed, some proc stats and also I have metadata catalog about each proc/process group with their custodians and business use case.

This data is then posted to sumologic for logging and also I trigger a series of notifications (Slack + PagerDuty hook to create an incident lifecycle).

I hope this helps

Upvotes: 1

Mike Thomsen
Mike Thomsen

Reputation: 37524

There's no universal way to append error messages as flowfile attributes. Also, we tend to strongly avoid anything like that because of the potential to bubble up error messages with sensitive data to users who might not be authorized to see those details.

Upvotes: 1

Related Questions