CODEWITHSUNDEEP
prometheussummarypromql
galmeida
galmeida

Reputation: 248

Is it possible to get the quantile in a time window in a summary metric across multiple servers?

Let's say I have n servers collecting an "elapsed time" (like response time) summary metrics for the 95% quantile for a given method call.

Can I get the value for this 95% quantile for all events occurred n the last x minutes? I know there is a problem with aggregating summaries from multiple sources, but I'm trying to extract something useful equivalent from it (it could be the maximum value for the quantile among the servers)

If I can, how would the query look like? Or, If I can only do it with one server, how would the query look like?

If I can't anyway, and I should use a histogram instead (I used a summary because the same component monitors calls with wildly varying expected durations, so I couldn't define bucket sizes that fit all) how would the query look like?

Upvotes: 0

Views: 1194

Answers (1)

valyala
valyala

Reputation: 17830

You can use max(summary_metric{quantile="0.95"}) for obtaining some kind of 95th quantile over multiple summary metrics. Obviously, the returned aggregate value may be far from the real 95th quantile, since quantile values cannot be aggregated in general case.

If you need calculating quantiles over multiple time series, then you have to use histograms instead of summaries. The following query can be used then for calculating 95th quantile estimation over multiple histograms over the last 5 minutes time window:

histogram_quantile(0.95, sum(increase(histogram_metric_bucket[5m])) by (le))

I used a summary because the same component monitors calls with wildly varying expected durations, so I couldn't define bucket sizes that fit all

This is a common issue with Prometheus histograms. There is a solution for this issue, which is called VictoriaMetrics histograms (I'm the author of VictoriaMetrics). These histograms automatically define the needed buckets, so users don't need to decide which bucket configuration to use.

Upvotes: 1

Related Questions