CODEWITHSUNDEEP
timeexecutionapache-nifi
Swati Sood
Swati Sood

Reputation: 245

apache nifi total execution time

I would like to know how to find out/capture the total execution time for any flow in Apache Nifi. Is there a way to do that and add it to the attributes list so that same can be shared over the PutEmail?

Upvotes: 7

Views: 6267

Answers (2)

Mark Payne
Mark Payne

Reputation: 191

James provided an excellent explanation above. One additional note, though, is that rather than adding an attribute for the timestamp of when the data was received, you can simply reference the lineageStartDate attribute:

${now():toNumber():minus(${lineageStartDate}):format("HH:mm:‌​ss")}

This is the same value that is used to determine the Lineage Duration in the Provenance.

Upvotes: 11

James
James

Reputation: 11931

I believe you would have to add a custom timestamp property when the flowfile is received, do the bulk of your processing, then calculate an elapsed time attribute to include in the email. You can do this with two UpdateAttribute processors.

  1. UpdateAttribute, received = ${now():toNumber()}

  2. (do processing)

  3. UpdateAttribute, elapsed = ${now():toNumber():minus(${received}):format("HH:mm:ss")}

This will format the elapsed time like "00:04:16" (4 minutes, 16 seconds). You can use this as ${elapsed} in your PutEmail message content.

But this is a bit ugly and only gives an approximation of the processing time. The NiFi provenance system maintains "Lineage Duration", which describes the elapsed time since the file entered NiFi. This is a far more authoritative number. But I do not believe you can query lineage duration from expression language. You would have to separately query and analyze the provenance data.

enter image description here

Upvotes: 12

Related Questions