CODEWITHSUNDEEP
prometheus
Dus
Dus

Reputation: 4240

Prometheus : how do i sum by with 2 different metrics

I have 2 different metrics : metric_a with a field type metric_b with a field type (same one)

I'm trying to summarise a and b, of the same type. If type exists only on metric_a and not on metric_b - it should return metric_b's result. I've tried a lot of options on prometheus:

sum by (type)(metric_a{job=~"provision-dev"}) or vector(0) + sum by(type)(metric_b{job=~"provision-dev"}) or vector(0) : returns only the values from metric_a, and doesn't calculate metric_b's results.

sum by (type)(metric_a{job=~"provision-dev"}) + sum by(type)(metric_b{job=~"provision-dev"}) : returns only the values from metric_b, and doesn't calculate metric_a's results.

sum by (cluster_id)(provision_scale_out_failures{job=~"provision-dev"} + provision_scale_out_success{job=~"provision-dev"}) : well this isn't even a right query

Basically here's an example of a success :

metric_a :

metric_b :

result of the query :

Upvotes: 20

Views: 53519

Answers (2)

valyala
valyala

Reputation: 18056

The following PromQL query should sum metric_a and metric_b by type:

(sum(metric_a) by (type) + sum(metric_b) by (type))
or
(sum(metric_a) by (type) unless sum(metric_b) by (type))
or
(sum(metric_b) by (type) unless sum(metric_a) by (type))

How it works:

  • The sum(metric_a) by (type) + sum(metric_b) by (type) sums time series with matching type label values on both sides of + according to matching rules
  • The sum(metric_a) by (type) unless sum(metric_b) by (type) returns sum(metric_a) by (type) results for type label values missing in sum(metric_b) by (type). See docs about unless operator.
  • The sum(metric_b) by (type) unless sum(metric_a) by (type) returns sum(metric_a) by (type) results for type label values missing in sum(metric_a) by (type).

Then results from these three queries are joined with or operator.

This query is equivalent to the query proposed by Michael: sum({__name__=~"metric_a|metric_b"}) by (type) .

P.S. This query can be simplified further when using MetricsQL:

sum(metric_a, metric_b) by (type)

This query works, since sum() function in MetricsQL accepts and sums arbitrary number of arguments.

Upvotes: 10

Michael Doubez
Michael Doubez

Reputation: 6903

This is the expected behavior when using a binary operator: both side must have a matching label set to be taken into account.

If you want to be able to aggregate both side and get the single one, you first must get the union of different metrics using the __name__ label:

 sum by(__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"})

You can cascade the aggregation operator:

sum by (type) (sum by (__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"}))

Finally, you can also compact everything into:

sum by (type) ({__name__=~"metric_a|metric_b",job=~"provision-dev"})

Upvotes: 17

Related Questions