CODEWITHSUNDEEP
counterprometheusgrafana
pete19
pete19

Reputation: 53

Time series from Prometheus source: how to set nulls as zero?

Working in Docker Grafana 8.1.5. Using time series graph, I'm plotting a Prometheus Counter source (that has one label) as a time series (by label), and need to fill all null/missing values as zeros.

This is the query applied to the Prometheus counter source, plotting the label code:

my_metric{code!=""}

The graph display works (just need to see the current counter value for each label variant, and the difference within the selected time range), but the new Grafana time series graph is missing an option that the Graph (old) has under Display > Stacking and null value > null value: null as zero, hence it now ends up with broken lines when null values occur.

Unfortunately, I cannot use the Graph (old) chart as I need the legend value difference, which only is available in the new time series graph.

I tried to add or on() vector(0) to the end of the query, but the condition does not get applied to the data series for each label variant, it rather adds a new data series all filled with zeros...

Thanks for any suggestions !

Upvotes: 11

Views: 46325

Answers (4)

Michal Kazmierczak
Michal Kazmierczak

Reputation: 51

I tried to add or on() vector(0) to the end of the query, but the condition does not get applied to the data series for each label variant

I understand that you need, sort of, backfill the subtractor (second) vector for all current label variants.

Following I find the easiest trick to achieve that:

my_metric - ((my_metric offset 1d) or my_metric * 0)

Grafana version of it:

my_metric - ((my_metric offset $__range) or my_metric * 0)

(do the instant query)

The trick is that the subtractor vector is built with the or operator. This results in a vector containing label sets and values from my_metric offset 1d, along with all label sets with zeroed values from the current set of labels.

Upvotes: 2

Evan Q
Evan Q

Reputation: 138

if you want to default 0 with labels, you can do it:

solution(promql)

my_metric_1{aaa="your_filter_val1",bbb="your_filter_val2"} or group by (aaa,bbb) ({aaa="your_filter_val1",bbb="your_filter_val2"}) (my_metric_2) < bool 0

background

my_metric_1{aaa="a", bbb="b"}

the metric can be null,

my_metric_2{aaa="a", bbb="b"}

the metric always existed.

explain

vector_with_labels(can nil) or vector2_with_labels(always 0)

others

maybe it is easy if usevictoriametrics,

Upvotes: 1

valyala
valyala

Reputation: 17830

The following technique can be used for filling gaps in a single time series returned from the query q:

sum(q) or vector(0)

The q is wrapped into sum() in order to drop all the labels for a time series returned from q, so the time series can be matched with vector(0) time series according to matching rules for or operator. The query without sum(): q or on() vector(0) would return two time series on the graph instead of a single time series: one time series from q with its own set of labels and another time series with zero labels and with zero value where the q time series contains gaps.

Unfortunately Prometheus doesn't provide an easy ability to fill gaps with zeroes if q returns multiple time series with distinct labelsets. Other Prometheus-like solutions such as VictoriaMetrics (I work on it) provides default operator for this case. For example, the following MetricsQL query fills gaps with zeroes for all the time series returned from q:

q default 0

Upvotes: 14

Felipe
Felipe

Reputation: 7563

I had this issue as well and I was not able to use only or on() vector(0) as you mentioned because the main query was returning NaN. In my case I had a division by zero.

I could get around of it by first evaluate if the query has a value >= 0 and then use the or on() vector(0). Try something similar to:

((my_metric{code!=""}) >= 0) OR on() vector(0)

Upvotes: 23

Related Questions