CODEWITHSUNDEEP
prometheus
Eran H.
Eran H.

Reputation: 1219

Prometheus rate query over a long period

When using the rate function over a long period (e.g. 7d) I am getting the error "query processing would load too many samples into memory in query execution".

My query is

histogram_quantile(0.90, rate(http_request_in_seconds_bucket[7d]))

Upvotes: 0

Views: 1344

Answers (1)

Eran H.
Eran H.

Reputation: 1219

This error happens because Prometheus has a limit of samples it can process in memory.

I solved this with subqueries which was added on Prometheus 2.7. This allows you to separately query smaller time intervals and then aggregate them together.

For example, I changed my query to multiple 24 hours subqueries and then averaged it together

histogram_quantile(0.90, avg_over_time(rate(http_request_in_seconds_bucket[24h])[7d:12h]))

There are other solutions:

  1. Increase the number of samples it can process by changing the flag query.max-samples (not recommended)
  2. Use Recording Rules - recommended over subqueries as it reduces the load on prometheus servers, but takes more efforts to set up.

Upvotes: 1

Related Questions