CODEWITHSUNDEEP
grafanaprometheus
bm1729
bm1729

Reputation: 2375

Time since a value was zero

I have an application that consumes work to do from an AWS topic. Work is added several times a day and my application quickly consumes it and the queue length goes back to 0. I am able to produce a metric for the length of the queue.

I would like a metric for the time since the length of queue was last zero. Any ideas how to get started?

Upvotes: 0

Views: 765

Answers (2)

bm1729
bm1729

Reputation: 2375

Similar to Alin's answer; upon revisiting this problem I found this from the Prometheus documentation:

https://prometheus.io/docs/practices/instrumentation/#timestamps,-not-time-since

If you want to track the amount of time since something happened, export the Unix timestamp at which it happened - not the time since it happened. With the timestamp exported, you can use the expression time() - my_timestamp_metric to calculate the time since the event, removing the need for update logic and protecting you against the update logic getting stuck.

Upvotes: 0

Alin Sînpălean
Alin Sînpălean

Reputation: 10064

Assuming a queue_size gauge that records the size of the queue, you can define a recorded rule like this:

# Timestamp of the most recent `queue_size` == 0 sample; else propagate the previous value
- record: last_empty_queue_timestamp
  expr: timestamp(queue_size == 0) or last_empty_queue_timestamp

Then you can compute the time since the last time the queue was empty as simply as:

timestamp(queue_size) - last_empty_queue_timestamp

Note however that because this is a gauge (and because of the limitations of sampling), you may end up with weird results. E.g. if one work item is added every minute, your sampling interval is one minute and you sample exactly after the work items have been added, your queue may never (or very rarely) appear empty from the point of view of Prometheus. If that turns out to be an issue (or simply a concern) you may be better off having your application export a metric that is the last timestamp when something was added to an empty queue (basically what the recorded rule attempts to compute).

Upvotes: 1

Related Questions