CODEWITHSUNDEEP
prometheusgrafanagrafana-alerts
Abhi Rathod
Abhi Rathod

Reputation: 11

Prometheus Increase function alert

So my goal is to set up alerting mechanism for network interface drops. I am using node_network_receive_drop_total which is a counter. My goal is to alert when drops occur meaning when count of drops go up. Lets say I have these values for drops over a period of time.

|node_network_receive_drop_total |difference
|0                               |0
|1                               |1
|5                               |4
|5                               |0  
|7                               |2

In this example, how can I use increase function and alert condition in grafana to alert when the difference is non-zero?

Upvotes: 1

Views: 5541

Answers (2)

Tomy8s
Tomy8s

Reputation: 91

In the Query tab a graph with a name something like node network drops and query something like this: increase(node_network_receive_drop_total[5m]).

When there are no drops, this graph should have a flat line at 0. When there is a drop the graph will show a line at 1 for 5 mins after the drop.

In the Alert tab create an alert with condition of WHEN max() OF query(A, 15m, now) IS ABOVE 0.

As you will only have one value returned by the query you could use max(), min(), or max() (they will all return the same value).

The A in query should match the letter on your query in the Query tab. If you have more than one query displayed on the graph, you may need to change this.

15m and now look at data from the last 15m.

Upvotes: 0

Sergio Santiago
Sergio Santiago

Reputation: 1514

In your example, you should use delta function. Something similar to:

delta(node_network_receive_drop_total[5m]) > 0

That means, whenever you have a delta greater than 0 in the last 5 minutes, your alert will be triggered.

PS: consider using sum in case you have multiple instances.

Upvotes: 3

Related Questions