CODEWITHSUNDEEP
prometheusprometheus-alertmanager
Deyan Farrugia
Deyan Farrugia

Reputation: 17

Prometheus 'labels'

I need to send a specific alert to multiple channels on Slack through alertmanager via a prometheus .yml file.

At the moment my current config is one to one, meaning one alert to one channel. Therefore the alert ' vs_replica_sql_slave_status' goes to 'slack_monitoring_prod'

prometheus.yml:

groups:
      - name: vs_replica_sql_slave_status
        rules:
           - alert: vs_slave_status
             for: 2m
             expr: (mysql_global_status_slave_running{instance=~"vs-replica.+",alias!~"vs-replica-test",alia
             labels:
               severity: "critical"
             annotations:
                     identifier: "{{ $labels.alias }}"
                     description: "Slave Status not running"

alertmanager.yml:

 routes:
  - match:
      severity: critical
    receiver: slack_monitoring_prod

I need to send the alert to another channel as well which is:

- match:
      severity: critical_dwh
    receiver: critical_dwh

In the first block of code I have labels > severity > critical which points to first channel. Is it possible to add another label that will point to the second channel or will this break the whole config? i.e:

labels:
   severity: "critical"
   severity: "critical_dwh"

Any suggestion or methods on how to perform this would be greatly appreciated.

Upvotes: 0

Views: 807

Answers (2)

Moein Tavakoli
Moein Tavakoli

Reputation: 96

you must add the two rules with diffrent severity to use in routing in alertmanager then in alertmanager you route the every severity to specific channel like this

aleertmanager.yml

routes:
  - match:
      severity: critical
    receiver: slack_monitoring_prod

  - match:
      severity: critical_dwh
    receiver: critical_dwh

receivers:
  - name: slack_monitoring_prod
    slack_configs:
      - api_url: 'your_slack_webhook_url'
        channel: '#slack_monitoring_prod'
        send_resolved: true

  - name: critical_dwh
    slack_configs:
      - api_url: 'your_slack_webhook_url'
        channel: '#critical_dwh'
        send_resolved: true

Upvotes: 0

thisisfine
thisisfine

Reputation: 11

You cannot have two labels with the same key. Instead you could add an additional label:

labels:
  severity: critical
  dwh: critical

Then you can route each label to different receivers. Remember to use continue: true for the pipeline to continue matching if you want the alert to reach multiple receivers.

route:
  routes:
  
  - receiver: slack_monitoring_prod
    matchers:
    - severity = critical
    continue: true
  
  - receivers: dwh_monitoring_prod
    matchers:
    - dwh = critical

You can use amtool (comes with Alertmanager) to verify your routing.

amtool config routes test --config.file /path/to/alertmanager.yml severity=critical
amtool config routes test --config.file /path/to/alertmanager.yml dwh=critical

Upvotes: 1

Related Questions