CODEWITHSUNDEEP
restapache-nifi
happy
happy

Reputation: 2628

Get source/upstream connection's processor name in nifi

I want to monitor the flowfiles in Nifi from business perspective.

So I have added executescript processor using python script which creates the message and pushes the same in elasticsearch after each processor.

I want the parent processor name or id of this executescript processor so that I will keep appending in the flowfile which will allow to know through which stages/processors this flow file is passed through and I can monitor it in ELK.

Upvotes: 1

Views: 1648

Answers (1)

Ben Yaakobi
Ben Yaakobi

Reputation: 1668

I think the best way to monitor FlowFiles is to use the Provenance logs. You can also export those logs to ELK using another NiFi instance and S2S.

Anyway, if you want to get the processor's name of a connection's source/destination using the REST API you can get it when you browse a process group's connections. Example:

/nifi-api/process-groups/{processGroupId}/connections/

You'll get an array of connections. In the connection object, you will get the name of the source in the path component/source/name. The same goes for destination.

EDIT:

To use provenance logs you need to do as follows:

  1. Send provenance logs to another NiFi instance(it is restricted to NiFi since it uses S2S).
  2. Parse the logs in this NiFi instance
  3. Send the logs to ElasticSearch using the PutElasticSearch5 processor.

It works the best and will help you monitor the FlowFiles the best :)

Upvotes: 2

Related Questions