CODEWITHSUNDEEP
prometheusmetrics
Daniel Bonkowski
Daniel Bonkowski

Reputation: 71

How to alert on metrics with a high cardinality in prometheus

when trying to create an alert on high metric cardinality with the expression count by(__name__) ({__name__=~".+"}) > 50 I get the error: vector contains metrics with the same labelset after applying rule labels.

As the expression works when using it directly in prometheus, I wonder if there is an actual way to use it in an alert?

Upvotes: 7

Views: 2108

Answers (1)

Jesko R.
Jesko R.

Reputation: 827

I think I found a solution for this issue, as I was trying it myself.

TL;DR

Use this promQL expression for alerting on metric cardinality:

label_replace(count by(__name__) ({__name__=~".+"}), "name", "$1", "__name__", "(.+)") > 50

Long Version

The issue as stated in the Prometheus error message. After the metric vector is converted to a vector of the alert, no labels differ and therefore are duplicated. This means

vector A ( metric_a{label=test}, metric_b{label=test} )

is converted in

vector B ( alert_a{label=test}, alert_a{label=test})

and that is why you have duplicates

( caveat: that is at least my understanding) By adding a new label with the metric name itself, you create a unique label set.

Upvotes: 1

Related Questions