CODEWITHSUNDEEP
prometheusprometheus-alertmanagerpromql
Abhishek Gayakwad
Abhishek Gayakwad

Reputation: 582

Prometheus grab series where label matches today's date

I have series which looks like this.

events{batch=20200818-00, stage=1} 10
events{batch=20200818-10, stage=1} 10
events{batch=20200818-20, stage=1} 10
events{batch=20200818-10, stage=2}  9
events{batch=20200818-20, stage=2}  8
events{batch=20200818-30, stage=2} 10

Where it has a label batch which is made of date and hour. I want to set an alert which triggers if total count of events for today in stage2 are grater than stage1

How can I filter the series based on time()? In Grafana it looks easy where you can interpolate the date and use regex to match label. But it look non trivial to set up and alert for such condition. Please help.

Upvotes: 1

Views: 5524

Answers (1)

Michael Doubez
Michael Doubez

Reputation: 6863

There is currently no way in Prometheus to cross the label/value. But you have a metric somewhere giving you the date of the day in a label and use it to filter your data.

Using a "today" metric

Generating it depends on your existing components: this could be done in a node_exporter textfile or in a recording rule you update everyday:

today{date="20200818"} 1

And using metric relabeling (either in configuration or using label_replace()),

label_replace(events, "date", "$1", "batch", "([0-9]+)-.*")

you obtain a similar label related to you metric:

events{batch=20200818-00, date=20200818, stage=1} 10

Then you can filter the series using :

label_replace(events, "date", "$1", "batch", "([0-9]+)-.*") AND ON(date) today

Avoid duplicated metrics

There is a small twist to take into account at midnight: the previous today metric will only disappear after 5 minutes when it becomes stale. You can work around that by generating yesterday's label but with a different value:

today{date="20200817"} 0
today{date="20200818"} 1

And the expression becomes:

label_replace(events, "date", "$1", "batch", "([0-9]+)-.*") AND ON(date) today == 1

NOTE: if you have a metric giving you the timestamp of the event, you can use it to determine what is today (this is what is usually done with batch)

Upvotes: 1

Related Questions