CODEWITHSUNDEEP
grafanaprometheuspromql
GavinE-OVO
GavinE-OVO

Reputation: 271

Simple cumulative increase in Prometheus

I have an application that increments a Prometheus counter when it receives a particular HTTP request. The application runs in Kubernetes, has multiple instances and redeploys multiple times a day. Using the query http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"} produces a graph displaying cumulative request counts per instance as is expected.

I would like to create a Grafana graph that shows the cumulative frequency of requests received over the last 7 days.

My first thought was use increase(...[7d]) in order to account for any metrics starting outside of the 7 day window (like in the image shown) and then sum those values.

I've come to the realisation that sum(increase(http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"}[7d])) does in fact give the correct answer for points in time. However, resulting graph isn't quite what was asked for because the component increase(...) values increase/decrease along the week.

How would I go about creating a graph that shows the cumulative sum of the increase in these metrics over the passed 7 days? For example, given the simplified following data

| Day | # Requests |
|-----|------------|
| 1   | 10         |
| 2   | 5          |
| 3   | 15         |
| 4   | 10         |
| 5   | 20         |
| 6   | 5          |
| 7   | 5          |
| 8   | 10         |

If I was to view a graph of day 2 to day 8 I would like the graph to render a line as follows,

| Day | Cumulative Requests |
|-----|---------------------|
| d0  | 0                   |
| d1  | 5                   |
| d2  | 20                  |
| d3  | 30                  |
| d4  | 50                  |
| d5  | 55                  |
| d6  | 60                  |
| d7  | 70                  |

Where d0 represents the initial value in the graph

Thanks

Upvotes: 27

Views: 10543

Answers (3)

Nicola Ambrosetti
Nicola Ambrosetti

Reputation: 2504

Using sum(increase(http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"}[$__interval])) as query is a good starting point and that's as far as Prometheus can go.

However, in order to get the cumulative graph, you need to leverage a "Transformation" in the Grafana UI as I'll describe below with a concrete example.

See for example this graph in the Grafana Playground

enter image description here

Let's select a single timeseries upper_95 that we want to plot the cumulative sum for:

enter image description here

Then select the "Transformations" tab and click "Add Transformation"

enter image description here

Choose the "Add field from calculation" transformation

enter image description here

Then choose Mode "Cumulative functions" and Calculation "Total" (should be the default). The documentation says:

Cumulative functions - Apply functions on the current row and all preceding rows. Total - Calculates the cumulative total up to and including the current row.

which is exactly what we want.

Since we have multiple timeseries we need to also select "upper_95" as Field, but if you only have one you can leave it as blank.

enter image description here

Now you have the cumulative sum graph available and you can select it in the legend to hide the other ones: enter image description here

Upvotes: 0

valyala
valyala

Reputation: 18094

Prometheus doesn't provide functionality, which can be used for returning cumulative increase over multiple time series on the selected time range.

If you still need this functionality, then try VictoriaMetrics - Prometheus-like monitoring solution I work on. It allows calculating cumulative increase over multiple counters. For example, the following MetricsQL query returns cumulative increase over all the time series with http_requests_total name on the selected time range in Grafana:

running_sum(sum(increase(http_requests_total)))

How does it work?

  1. It calculates increase per each time series with the http_requests_total name. Note that the increase() in the query above doesn't contain lookbehind window in square brackets. VictoriaMetrics automatically sets the lookbehind window to the step value, which is passed by Grafana to /api/v1/query_range endpoint. The step value is the interval between points on the graph.

  2. It sums increases returned at step 1 with the sum() function individually per each point on the graph.

  3. It calculates cumulative increase over per-step increases returned at step 2 with the running_sum function.

Upvotes: 3

nezzzzzz
nezzzzzz

Reputation: 122

If I understood your question's idea correctly, I think I managed to create such graph with a query like this

sum(max_over_time(counterName{someLabel="desiredlabelValue"}[7d]))

A graph produced by it looks like the blue one: Cumulatively processed during the last 7 days

The reasons why the future part of the graph decreases are both because the future processing hasn't obviously yet happened and because the more-than-7-days-old processing slides out of the moving 7-day inspection window.

Upvotes: 0

Related Questions