Which prometheus query function to monitor a rapid change of a counter?

Question

I have a prometheus counter that keeps track of the amount of documents processed in a scheduled job. I'm interested on plotting that counter on grafana.

Ideally I would have something like this, that shows the counter when the job was executed, but so far I was not able to.

Some solutions I have tested:

• using a gauge instead of a counter and set it to 0 at the end of the job. This doesn't work correctly in case of short lived jobs and in general the value of the gauge can become zero before that prometheus can scrape the API.
• using max_over_time: this doesn't go to zero
• using increase: this is promising, but you have to know the average duration of the job, and the graph just looks weird (see below)

Answer 1

Use rate/irate function - doc: https://prometheus.io/docs/prometheus/latest/querying/functions/#irate

Then you can also use sum for aggregating over time (if don't want per second rate).