CODEWITHSUNDEEP
prometheusprometheus-node-exporter
Patryk
Patryk

Reputation: 24130

Subtract rate and a sum of rates

Having a network transmit metric e.g. node_network_transmit_bytes_total from nodeexporter I'd like to get a difference between the transmit rate of an interface (enp3s0 in my case) and a sum of all bridge interface transmit rates.

I have something like this:

irate(node_network_transmit_bytes_total{device="enp3s0"}[1m]) -  irate(node_network_transmit_bytes_total{device=~"br.*"}[1m])

but this gives me no datapoints.

EDIT

I've tried what Prometheus/PromQL subtract two gauge metrics suggests but my case is a bit different because on the right hand side I have more than 1 series.

So this:

node_network_transmit_bytes_total{device="enp3s0"} - ignoring(device) node_network_transmit_bytes_total{device=~"br.*"}

yields:

Error executing query: found duplicate series for the match group {instance="192.168.X.Z:9100", job="nodeexporter"} on the right hand-side of the operation: [{name="node_network_transmit_bytes_total", device="br-XXXX", instance="192.168.X.Z:9100", job="nodeexporter"}, {name="node_network_transmit_bytes_total", device="br-5d6dce95c2b0", instance="192.168.X.Z:9100", job="nodeexporter"}];many-to-many matching not allowed: matching labels must be unique on one side

I've tried using sum():

node_network_transmit_bytes_total{device="enp3s0"} - ignoring(device) sum(node_network_transmit_bytes_total{device=~"br.*"})

but that yields no results again.

EDIT2

I've managed to figure out how to get a difference by

sum(node_network_transmit_bytes_total{device="enp3s0"}) by (instance) - sum (node_network_transmit_bytes_total{device=~"br.*"}) by (instance)

but I cannot use irate on it:

irate(sum(node_network_transmit_bytes_total{device="enp3s0"}) by (instance) - sum (node_network_transmit_bytes_total{device=~"br.*"}) by (instance))[5m]

Error executing query: 1:149: parse error: ranges only allowed for vector selectors

Upvotes: 2

Views: 4283

Answers (2)

valyala
valyala

Reputation: 18010

First of all, it is recommended to use rate() instead of irate(), since irate() tends to return jumpy results - see this article for details.

If you want to calculate per-instance sum of network transmit rates for devices with names starting with br, then the following query must be used:

sum(rate(node_network_transmit_bytes_total{device=~"br.*"}[1m])) by (instance)

See sum() docs for details.

The rate(node_network_transmit_bytes_total{device="enp3s0"}[1m]) returns per-instance network transmit rates for enp3s0 device. It is likely the returned time series contain additional labels other than instance label. So the following query returns empty results because of time series matching rules for binary operators in Prometheus:

rate(node_network_transmit_bytes_total{device="enp3s0"}[1m])
  -
sum(rate(node_network_transmit_bytes_total{device=~"br.*"}[1m])) by (instance)

This can be fixed in two ways:

  1. To wrap the rate(node_network_transmit_bytes_total{device="enp3s0"}[1m]) into sum(...) by (instance). This effectively removes all the labels other than instance from result and allows finding time series pairs with identical sets of labels on the left and the right side of -:
sum(rate(node_network_transmit_bytes_total{device="enp3s0"}[1m])) by (instance)
  -
sum(rate(node_network_transmit_bytes_total{device=~"br.*"}[1m])) by (instance)
  1. To add on (instance) modifier after - operator:
rate(node_network_transmit_bytes_total{device="enp3s0"}[1m])
  - on (instance)
sum(rate(node_network_transmit_bytes_total{device=~"br.*"}[1m])) by (instance)

This will preserve all the original labels for node_network_transmit_bytes_total time series. See these docs for details on on() modifier.

Upvotes: 0

Patryk
Patryk

Reputation: 24130

OK it seems this was the solution:

(sum(irate(node_network_receive_bytes_total{device="enp3s0"}[1m])) by (instance))
 -
(sum(irate(node_network_receive_bytes_total{device=~"br.*"}[1m])) by (instance))

Upvotes: 3

Related Questions