CODEWITHSUNDEEP
prometheus
moffeltje
moffeltje

Reputation: 4658

How to subtract metrics with different labels?

I have two different metrics I need to calculate the difference for:

total and current{app}

The expression that works so far:

max total offset 30s - max current{app="myapp"}

This gives my a number that I use for an alert about the speed of some process for myapp.

However, I have multiple apps. So I want to get the difference for each app. But when I try to get a maximum by app, prometheus does not calculate the difference anymore because the two metrics have different labels:

Expression max total offset 30s - max by (app) (current)

Output {} (nothing)

And that is because the result of the individual calculations have different labels:

total{} and current{app="someapp"}

I was hoping there would be a prometheus function to add a label to total, to generate the same metric for all apps (which of course would already be a bit nasty).

How can I get an expression that returns the difference per app?

Expected result

{app="myapp"}    333
{app="someapp"}    9

Upvotes: 2

Views: 1033

Answers (1)

Michael Doubez
Michael Doubez

Reputation: 6863

If I understand correctly, you have a metric without label on the left hand side of the subtraction and multiple metrics on the right hand side.

This seems to be a case of one to many matching:

 max total offset 30s - on() group_right max by (app) (current)

The on() clause indicates that labels should be disregarded and group_right indicates the many side is the right hand side.

Upvotes: 3

Related Questions