CODEWITHSUNDEEP
prometheuspromql
blee
blee

Reputation: 606

Prometheus metric comparison alert

I'm trying to create an alert expression that fires if the given metric is not the same across all targets. Let's call my metric my_gauge where the expression console output shows:

my_gauge{group="test",instance="huey:9100",job="example"}   10
my_gauge{group="test",instance="duey:9100",job="example"}   10
my_gauge{group="test",instance="luey:9100",job="example"}   7

I'm interested in the target results that are below the highest threshold. So my initial thought was to compare it to the max (my_gauge) result which is:

{} 10

However my_gauge < max(my_gauge) returns no data (the expected result here would be luey). How do I write an expression that would return a target with a divergent metric?

Upvotes: 3

Views: 998

Answers (1)

Michael Doubez
Michael Doubez

Reputation: 6923

If you only need to alert on wether at least one value is different, you can compare if minimum value is the same as maximum value:

expr: min(my_gauge) by(group) != max(my_gauge) by(group)

If you want to alert on every value that differs, you can use the median to determine the common value (10 in your example). And then compare to it.

expr: my_gauge != on(group) group_left quantile(0.5, my_gauge) by(group)

Regarding your expression, you are missing the vector matches:

expr: my_gauge < on() group_left max(my_gauge)

Upvotes: 2

Related Questions