CODEWITHSUNDEEP
prometheusgrafana
GrafMetrics
GrafMetrics

Reputation: 61

Grafana prometheus set $__interval to 1s

I now have my $__interval on 2s because when I try setting it to 1s it won't display the query anymore. Why is 2s the lowest I can go?

irate(rx_bytes{instance="localhost:2000"}[2s]) works perfectly fine

irate(rx_bytes{instance="localhost:2000"}[1s]) query just disappears

Upvotes: 1

Views: 1506

Answers (1)

valyala
valyala

Reputation: 18094

Prometheus calculates irate(m[d]) at a timestamp t in the following way:

  1. It selects raw samples on a time range (t-d .. t] for each series with the name m. Note that t-d isn't included in the time range.
  2. It finds the last two samples - (t1, v1) and (t2, v2) on the selected time range per each matching series, where t1 and t2 are timestamps and v1 and v2 are values for the last two samples on the selected time range.
  3. It calculates irate=(v2-v1)/(t2-t1) individually per each matching series.

As you can see, Prometheus can calculate irate if the selected time range contains at least two raw samples. Otherwise it returns empty result. So, it is likely the interval between samples in your case (aka scrape_interval) is greater or equal to 1s. So any time range (t-1s .. t] contains less than two samples.

The workaround is to use VictoriaMetrics (the project I work on). This is a Prometheus-like monitoring solution, which takes into account the previous raw sample just before the selected time range when calculating irate and rate, so it returns the expected result for small lookbehind windows in square brackets. See this article for details.

P.S. it is recommended to use rate() instead of irate() in most cases, because irate() doesn't capture spikes - it just performs calculations based on an arbitrary set of raw samples. See this article for details.

Upvotes: 2

Related Questions