CODEWITHSUNDEEP
alertmetricsgrafanaprometheus
Alexander Rainchik
Alexander Rainchik

Reputation: 73

How to show oldest metrics from queue? Grafana/Prometheus

I have a panel in grafana, which shows current alerts (simple query ALERTS{alertstate="firing"} with instant=enable option). I want to know the time, when alert triggered in first time. How can I do this?

As I think, I need to sort queue for Time values and take the oldest. But I can't find tools for this.

Upvotes: 3

Views: 1791

Answers (1)

Alin Sînpălean
Alin Sînpălean

Reputation: 10084

You could start with something like this query:

timestamp(
    absent(ALERTS{env=~"$env",job=~"$job",alertstate="firing"} offset 1m)
  unless
    absent(ALERTS{env=~"$env",job=~"$job",alertstate="firing"})
)

with Min step = 1m and Resolution = 1/1, but this will give you all the times any alert started to fire (regardless of whether it is still firing or not; and including every time it flipped on).

Or you could use a Discrete panel (which is what I'm doing) to eyeball when alerts started and stopped firing, and optionally when the alert went from warning to critical (if you set them up that way).

Upvotes: 1

Related Questions