Prometheus alertmanager send notifications to multiple slack channel

We have two different teams working on different applications.I would like send alert notifications to different slack channels via using same alerts expressions. I found some examples but not understand what is the main reason to use receiver: 'default' when try to add new route? What is the role of this and what if it affects if ı change this?

Meanwhile will be appreciate if you can help how should I send the notifations to multiple slack channels.. New one is what I tried.

Current alertmanager.yml

receivers:
  - name: 'team-1'
    slack_configs:
    - api_url: 'https://hooks.slack.com/services/1'
      channel: '#hub-alerts'
route:
  group_wait: 10s
  group_interval: 5m
  receiver: 'team-1'
  repeat_interval: 1h
  group_by: [datacenter]

New alertmanager.yml

alertmanager.yml:
    receivers:
      - name: 'team-1'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/1'
          channel: '#channel-1'
          send_resolved: true
      
      - name: 'team-2'
        slack_configs:
        - api_url: 'https://hooks.slack.com/services/2'
          channel: '#channel-2'
          send_resolved: true

route:
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 1h
  group_by: [datacenter]
  receiver: 'default'
  routes:
  - receiver: 'team-1'
  - receiver: 'team-2'

Upvotes: 16

Answers (5)

VictorNuzhdin

Reputation: 59

This works for me with your examples, guys 😃

##--Configuration v4.1 :: multiple receivers (Telegram + Gmail)
#
global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
    enable_http2: true
  smtp_hello: localhost
  smtp_require_tls: true

templates:
  - '/etc/alertmanager/_templates/*.tmpl'

route:
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 5m
  #
  ## default receiver (only one can be assigned)
  #receiver: blackhole
  receiver: gmail-notifications
  #receiver: tg-notifications
  #
  routes:
    - receiver: tg-notifications
      group_wait: 10s
      match:
        alertname: PrometheusTargetMissing
        severity: critical
      continue: true
    - receiver: gmail-notifications
      group_wait: 10s
      match:
        alertname: PrometheusTargetMissing
        severity: critical
      continue: true
  #
receivers:
  ## Sends Notifications to Null (discard)
  - name: blackhole
  #
  ## Sends Notifications to Telegram
  - name: tg-notifications
    telegram_configs:
    - send_resolved: true
      disable_notifications: false
      http_config:
        follow_redirects: true
        enable_http2: true
      api_url: https://api.telegram.org
      chat_id: <hidden>
      bot_token_file: "/etc/alertmanager/_secrets/sec_tg_token.txt"
      #
      # Custom message-template
      parse_mode: HTML
      message: '{{ template "telegram.default" . }}'
  #
  ## Sends Notifications to Gmail
  - name: gmail-notifications
    email_configs:
    - send_resolved: true
      require_tls: true
      from: <hidden>
      to: <hidden>
      hello: localhost
      smarthost: smtp.gmail.com:587
      auth_username: <hidden>
      auth_identity: <hidden>
      auth_password_file: /etc/alertmanager/_secrets/sec_smtp_pwd
      # Custom message-template
      headers:
        From: <hidden>
        To: <hidden>
        Subject: '{{ template "custom_mail_subject" . }}'
      html: '{{ template "custom_mail_html" . }}'

Prometheus v2.47.2 + AlertManager v0.26.0

I have the problem with this sythtax

  routes:
    - receiver: tg-notifications
      group_wait: 10s
      match_re:
        severity: "Critical|High"
      continue: true

In my case it didn't work and the subroutes didn't triggered, but only default route worked fine. I spending 3 days for read tons of examples that didn't works 😭

Upvotes: 2

ngougni heguens

Reputation: 1

I have this config but the alert are send just in one channel not the 2/

    config:
      global:
        
        resolve_timeout: 5m
      route:
        group_by: ['alertname', 'job', 'severity']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h
        receiver: "default"
        routes:

          - match:
              alertname: PreprodMonitor
              severity: warning|info  
            receiver: "default"
            continue: true

          - match:
              alertname: PreprodMonitor
              severity: critical
            receiver: "critical" 
            continue: true

                      
      receivers:
         # send to a slack channel. This is being used by the warning critical
        - name: "critical"
          slack_configs:
            - send_resolved: true
              api_url: https://hooks.slack.com/services/critical
              channel: '#kube-preprod2-alerting-critical'
              text: "{{ range .Alerts }}{{ .Annotations.message }}\n{{ .Annotations.runbook_url }}\n{{ end }}" 
              icon_url: https://avatars3.githubusercontent.com/u/3380462
              title: "{{ .CommonLabels.alertname }} - severity: {{ .CommonLabels.severity }}\n"
              # title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
            # send to a slack channel. This is being used by the warning severity
        - name: "default"
          slack_configs:
            - send_resolved: true
              api_url: https://hooks.slack.com/services/warning
              channel: '#kube-preprod2-alerting'
              # title: "{{ range .Alerts }}{{ .Labels.alertname }} - {{ .Labels.severity }}\n{{ end }}"
              text: "{{ range .Alerts }}{{ .Annotations.message }}\n{{ .Annotations.runbook_url }}\n{{ end }}" 
              icon_url: https://avatars3.githubusercontent.com/u/3380462
              title: "{{ .CommonLabels.alertname }} - severity: {{ .CommonLabels.severity }}\n"
              # te

Upvotes: 0

Sheikh Aafaq Rashid

Reputation: 199

Note! Make sure there is not any gap or newline between continue: true and next receiver.

# alertmanager.yml

route:
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 10s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 30s

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: "one"

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.
  routes:
    - receiver: "one"
      group_wait: 10s
      match_re:
        severity: "Critical|High|Medium|Low|warning|info"
      continue: true
    - receiver: "two"
      group_wait: 10s
      match_re:
        severity: "Critical|High|Medium"
      continue: true
    - receiver: "three"
      group_wait: 10s
      match_re:
        severity: "Critical|High"
      continue: true

receivers:
  - name: "one"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T2ND83CHL/B05G3RPUE03/kEFGLfmR0RtDVEsFUwrlkKv'
        send_resolved: true
        channel: 'one'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

  - name: "two"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T2ND83kuL/B05GGJEAV8R/LEEkDn24LE7BAVQr4HNWW76h'
        send_resolved: true
        channel: 'two'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

  - name: "three"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/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'
        send_resolved: true
        channel: 'three'
        icon_url: https://prometheus.io/assets/prometheus_logo_grey.svg
        title: |-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}}
          *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
          *Description:* {{ .Annotations.description }}
          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

Upvotes: 0

kachwa

Reputation: 29

I have alertmanagerconfig with below configuration and now I need to point info alerts to a null receiver, can I have multiple receiver and receivers?

kind: AlertmanagerConfig
metadata:
  name: Prometheus-alertmanager-config
  namespace: Prometheus
spec:
  route:
    receiver: alert-email-pagerduty-config
    groupBy: ['alertname', 'priority','severity']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 15m
    continue: true
  receivers: 
  - name: alert-email-pagerduty-config
    emailConfigs:
    - to: {{.to_email}}
      sendResolved: true
      from: {{.from_email}}
      smarthost:  {{.smarthost}}
      authUsername: {{.mail_username}}
      authPassword:
        name: 'alert-smtp-password'
        key: 'password'
      requireTLS: true
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: alert-smtp-password
  namespace: prometheus
stringData:
  password: {{.mail_password}}

Upvotes: 0

Saurabh Maurya

Reputation: 980

You need to set the continue property on your route to true. By default it is false.

The default behaviour of AlertManager is to traverse your routes for a match and exit at the first node it finds a match at.

What you want to do is fire an alert at the match and continue to search for other matches and fire those too.

Relevant documentation section: https://prometheus.io/docs/alerting/latest/configuration/#route

An example using this: https://awesome-prometheus-alerts.grep.to/alertmanager.html

In-lined the example above in case it ever breaks.

# alertmanager.yml

route:
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 10s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 30m

  # A default receiver
  receiver: "slack"

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.
  routes:
    - receiver: "slack"
      group_wait: 10s
      match_re:
        severity: critical|warning
      continue: true

    - receiver: "pager"
      group_wait: 10s
      match_re:
        severity: critical
      continue: true

receivers:
  - name: "slack"
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxxx'
        send_resolved: true
        channel: 'monitoring'
        text: "{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"

  - name: "pager"
    webhook_config:
      - url: http://a.b.c.d:8080/send/sms
        send_resolved: true

Upvotes: 29

Prometheus alertmanager send notifications to multiple slack channel

Answers (5)

Related Questions