CODEWITHSUNDEEP
grafanaprometheus
parliamentowl
parliamentowl

Reputation: 319

Grafana Singlestat Max not matching Graph with same query

I have a Singlestat panel and Graph panel that use an identical query, (Singlestat & Graph query). And, the Singlestat is set to max (Singlestat setting).

Unfortunately, the graph clearly shows a maximum greater than the max singlestat (714 vs ~800): Singlestat vs Graph. Judging from the sparklines on the Singlestat, it seems like the Singlestat's calculations are less granular than the graph's. Can anyone explain why this would be if they're using the same base query? The other singlestat functions (like Min, Avg, etc.) seem to work fine. It's just max that I'm seeing this issue with.

Note: I reviewed the other Grafana Singlestat vs Graph posts, but this appears to be a different issue.

Upvotes: 1

Views: 1357

Answers (1)

Alin Sînpălean
Alin Sînpălean

Reputation: 10134

If you take a look at the first image you linked to, you'll notice there is a Min step input, with a default value of 5m. That's where your lower resolution comes from. You may set that explicitly to your scrape interval (or less, to make sure you don't lose any samples due to jitter in the scrape interval, although that may end up being costly), but if you increase your dashboard range enough you'll:

  • (a) likely have a singlestat max value that's higher than anything on the graph (because your graph is now lower resolution than the singlestat source data); and
  • (b) will hit Prometheus' 11K samples limit if you zoom out to a range longer than 11K times the scrape interval.

Your best bet is to use PromQL to calculate the max value to display in your singlestat panel. You'll still have to deal with (a) above (low resolution graph when the range is long) but it's going to be the actual max (as much as the fact that you're actually sampling values at some fixed interval allows) and it's going to be more efficient.

Problem is that given your query -- sum(jvm_thread_count) -- there is no way of putting that into a single PromQL query with max_over_time. You'd have to define a recorded rule (something like instance:jvm_thread_count:sum = sum(jvm_thread_count) and then have your singlestat panel display the results of the max_over_time(instance:jvm_thread_count:sum[$__range_s]) instant query (check the Instant checkbox in your singlestat settings).

Upvotes: 2

Related Questions