CODEWITHSUNDEEP
prometheuspromql
Prasanth Ravi
Prasanth Ravi

Reputation: 155

Subtract two metrics across a range and then find the average

I'm trying to subtract two metrics (guages) with the same lables over a time period [5m], and then average the result.

Queries I have tried,

avg_over_time(jvm_memory_bytes_max{area="heap", app="ui",kubernetes_pod_name="ui-dep-76b4f95bf4-xcs4s"}[5m] -  jvm_memory_bytes_used{area="heap", app="ui",kubernetes_pod_name="ui-76b4f95bf4-xcs4s"}[5m]) 

avg_over_time(jvm_memory_bytes_max{area="heap", app="ui",kubernetes_pod_name="ui-dep-76b4f95bf4-xcs4s"} -  jvm_memory_bytes_used{area="heap", app="ui",kubernetes_pod_name="ui-76b4f95bf4-xcs4s"})[5m]

avg_over_time((jvm_memory_bytes_max - on(area, app, kubernetes_pod_name) jvm_memory_bytes_used)[5m])

All of these have parse errors because the query is not correct.

Most of the example over ranges use avg_over_time to reduce a metric to a single value and then use an arithmetic operation.

I'm not sure if it's applicable because I don't want to average over a metric and then apply an arithmetic function. But rather, I want to apply the arithmetic operation on two data points of different metrics and then average the results.

Is that possible ? If so, how would I go about it ?

Upvotes: 0

Views: 2625

Answers (2)

valyala
valyala

Reputation: 18056

Try the following query:

avg_over_time((jvm_memory_bytes_max{area="heap", app="ui",kubernetes_pod_name="ui-dep-76b4f95bf4-xcs4s"} -  jvm_memory_bytes_used{area="heap", app="ui",kubernetes_pod_name="ui-76b4f95bf4-xcs4s"})[5m:10s])

It calculates the difference of two metrics on every 10-second interval and then calculates the average over the differences for the last 5 minutes. The query uses Prometheus subquery feature.

Upvotes: 1

Ali Sattari
Ali Sattari

Reputation: 334

One way to have this working is to have the inner expression as a recording rule then apply avg_over_time to that for whatever range you like. As sub-queries are still very limited in PromQL using recording rules to create metrics from expressions to be used in other expressions is a common solution.

In you case it would be like:

...
  - name: memory_diff
    rules:
    - record: jvm_memory_diff_max_used
      expr: jvm_memory_bytes_max - jvm_memory_bytes_used
...

Then:

avg_over_time(jvm_memory_diff_max_used[5m])

Upvotes: 1

Related Questions