CODEWITHSUNDEEP
kubernetesprometheus
Priyanka
Priyanka

Reputation: 82

Prometheus nested alert query in prometheus

Can it b possible to add nested query in prometheus alerts? I am using prometheus to monitor kubernetes cluster.

Alert is generated if node is down but i want to configure alert so that if any node is down then prometheus should not send alert for pods and services which are running on that.

Something like this,

if(pod_down)
  if(corresponding_node_down)
    //dont send alert
    //node down alert is in firing state

Upvotes: 0

Views: 896

Answers (2)

Christian Will
Christian Will

Reputation: 1695

Inhibition

Take a look at this Prometheus Alertmanager configuration.

link: prometheus.io/docs/alerting/configuration/#inhibit_rule

Inhibition is a concept of suppressing notifications for certain alerts if certain other alerts are already firing.

Example: An alert is firing that informs that an entire cluster is not reachable. Alertmanager can be configured to mute all other alerts concerning this cluster if that particular alert is firing. This prevents notifications for hundreds or thousands of firing alerts that are unrelated to the actual issue.

reference: https://prometheus.io/docs/alerting/alertmanager

Upvotes: 2

brian-brazil
brian-brazil

Reputation: 34152

If a node is down, then Kubernetes should be handling that automatically and moving services elsewhere. Accordingly an alert like this isn't very useful.

What I'd suggest you do it alert on user-visible symptoms such as high latency and error ratios rather than individual causes such as a machine or container being down.

Upvotes: 2

Related Questions