CODEWITHSUNDEEP
loggingapache-nifi
Cribber
Cribber

Reputation: 2943

Error logging in NiFi with PutSQL processors

I want to insert SQL rows into my_logging_table as a log whenever a NiFi Put-SQL-processor in my pipeline fails. So far I had to create another logging-PutSQL processor for every pipeline-processor which have the following sql-statement:

insert into schema.my_logging_table values
('Failed-NiFi-processor-name', 'failed', current_timestamp)

This obviously leads to double the number of NiFi processors as each has to have their own "logging-putsql-processor" so that the correct processor name can be logged.

Is there a way to have my pipeline-putsql-processors update a flowfile property on fail (Im thinking of passing on the name of the processor)?

This way I could route all failures to a single logging-putsql-processor which reads the failing processor's name from the file property and insert the row into the database. I noticed the "update_attribute" processor, but I would have to build one of them for every processor as well...

Upvotes: 2

Views: 1148

Answers (1)

Sdairs
Sdairs

Reputation: 2032

Yes, you'd need to stick an UpdateAttribute on the Failure branch, not all processors write failure attributes. You'll see what attributes they write in the processor docs - see the Writes Attributes section for PutSQL

Alternatively, you could

  1. Create a flow to parse nifi-app.log and hunt for the errors
  2. Create a SiteToSiteBulletinReportingTask flow that captures all bulletins, sends them to an RPG, filters for PutSQL failures, sends to DB, etc.

Upvotes: 1

Related Questions