Prometheus & grafana: wrong total in legend

Question

I have a metric with amount of tasks picked from queue. I use this query for prometheus in graphana:

sum by (namespace) (increase(dbqueue_picked_tasks_total{service="$service", namespace="$environment"}[1m]))

For choosen time there was only 1 task picked from queue. But on graph I see 3 points with values 1.2 and in legend I see total 3.6. What I'm doing wrong?

Answer 1

Sometimes it is necessary to change the focus of the mouse from the query text field in Grafana to another place just to update the chart.

In your case, the Total legend option from Grafana is summing up the points based on the time frame [1m] of the Prometheus query. Use the Current legend option on Grafana to see the last value. Or use the Max option to see the unique maximum value from the whole timeseries.