CODEWITHSUNDEEP
google-cloud-platformprometheusgrafanapromqlgoogle-managed-prometheus
Matanya
Matanya

Reputation: 213

Why do panels with Google Cloud Monitoring in Grafana vary widely between refreshes?

We use Google Managed Prometheus in our GKE cluster. I'd like to use the Google Cloud Monitoring Grafana plugin if possible (using GMP as a Prometheus data source needs essentially a shim deployed somewhere). Using the PromQL querier for Google Cloud Monitoring works, but every time the panel reloads, I get completely different values. This is the query I'm running:

sum(increase(some_counter_metric[1y])) / 1000000

This gives me essentially a number, but in my case, it can swing from ~50 to ~200 and back every 5 seconds.

Similar behavior crops up with other graphs using PromQL and Google Cloud Monitoring in Grafana.

Upvotes: -1

Views: 191

Answers (1)

Dion V
Dion V

Reputation: 824

You may consider checking the query syntax again for any typos or incorrect operators.

Cloud Monitoring typically stores data at a coarser resolution than Prometheus (by default, 1m). You can try as well to adjust it to [5m] or [12m] to see if it stabilizes. Also Cloud Monitoring stores data by aggregation period, it might be some timestamps were missing. You can try to use interpolation functions to estimate missing values like linear or holt\_win\_linear.

Lastly, you can try to update Grafana to the latest. There are times updates to the latest version resolves issues or bugs known.

Upvotes: 0

Related Questions