CODEWITHSUNDEEP
grafanaprometheus
SangminKim
SangminKim

Reputation: 9136

Why does gauge type of metric value keep increasing in node-exporter of prometheus?

This is a snippet of node-exporter metrics.

# HELP node_network_receive_bytes Network device statistic receive_bytes.
# TYPE node_network_receive_bytes gauge
node_network_receive_bytes{device="br-074eb8733fdc"} 4.5000969e+07
node_network_receive_bytes{device="br-d24ce0793158"} 9.8563483e+07
node_network_receive_bytes{device="docker0"} 1.81893686701e+11
node_network_receive_bytes{device="eno1"} 1.30390371207e+11
node_network_receive_bytes{device="eno2"} 2.7347435325e+10
node_network_receive_bytes{device="lo"} 9.80764398145e+11
node_network_receive_bytes{device="veth9eee40a"} 9.5458576e+07
node_network_receive_bytes{device="vethb89d9df"} 1.2443436876e+11
node_network_receive_bytes{device="vethd5ca4a4"} 648

It is saying the type of node_network_receive_bytes is gauge (frankly, I not sure if this is the right way to check metric type but looks right intuitively).

However, when I checked node_network_receive_bytes with range-vector, it shows figures keeping increasing such as counter type.

node_network_receive_bytes{device="eno1"}[3m]

    130393948462 @1516931391.405
    130394168285 @1516931401.405
    130394376002 @1516931411.405
    130394579742 @1516931421.405
    130394755152 @1516931431.405
    130394955813 @1516931441.405
    130395174828 @1516931451.405
    130395475287 @1516931461.405
    130395734293 @1516931471.405
    130395935167 @1516931481.405
    130396110667 @1516931491.405
    130396314762 @1516931501.405
    130396490334 @1516931511.405
    130396675817 @1516931521.405
    130396825764 @1516931531.405
    130397011068 @1516931541.405
    130397158242 @1516931551.405
    130397367815 @1516931561.405

In addition, Grafana dashboard which I downloaded for node-exporter queries this metric using irate and increase which is for counter type metric as I know.

// Query in grafana dashboard for node-exporter
sum(irate(node_network_receive_bytes{device=~"$device",instance=~"$node"}[3m]))
sum(increase(node_network_receive_bytes{device=~"$device",instance=~"$node"}[1m]))

increase()

increase should only be used with counters. It is syntactic sugar for rate(v) multiplied by the number of seconds under the specified time range window, and should be used primarily for human readability. Use rate in recording rules so that increases are tracked consistently on a per-second basis.

irate()

irate should only be used when graphing volatile, fast-moving counters. Use rate for alerts and slow-moving counters, as brief changes in the rate can reset the FOR clause and graphs consisting entirely of rare spikes are hard to read.

I have been so confused, what I have missed? (node_network_transmit_bytes also appears the same symptom.)

Upvotes: 0

Views: 1483

Answers (1)

brian-brazil
brian-brazil

Reputation: 34112

These are actually counters, and in version 0.16.0 will have the correct type and metric name.

The node exporter is one of the oldest exporters and has some cruft built up from before all the guidelines were established.

Incidentally, it is valid for a gauge to appear to be monotonically increasing. With a gauge though you care about its absolute value, with a counter only its rate of increase.

Upvotes: 1

Related Questions