CODEWITHSUNDEEP
kubernetesprometheusprometheus-alertmanager
DmitrySemenov
DmitrySemenov

Reputation: 10305

automatic labels to prometheus alertmanager rules

I'm using prometheus-community/prometheus chart

I'd like to add the following labels automatically to any alert manager rule firing

so that I don't these labels manually to each alert rule.

 - alert: NGINXTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4.+"} ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
     env: prod
     cluster: project-prod-eks              <---------------HOW to inject them?
   annotations:
     description: Too many 4XXs
     summary: More than 5% of all requests returned 4XX, this requires your attention

so that I can do something like

 - alert: NGINXTooMany400s
   expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4.+"} ) / sum(nginx_ingress_controller_requests) ) > 5
   for: 1m
   labels:
     severity: warning
   annotations:
     description: Too many 4XXs on {{ $labels.env }} / {{ $labels.cluster }}  <----- THIS
     summary: More than 5% of all requests returned 4XX, this requires your attention

Any ideas?

Upvotes: 1

Views: 6529

Answers (2)

anemyte
anemyte

Reputation: 20176

You can add external_labels to your prometheus.yml:

global:
  # The labels to add to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    env: prod
    cluster: project-prod-eks

The community chart has it in values.yml:

serverFiles:
  prometheus.yml:
    global:
      external_labels:
        foo: bar
    ...

Upvotes: 4

DmitrySemenov
DmitrySemenov

Reputation: 10305

so I did slightly differently that modifying the "serverFiles", see below

server:
  nodeSelector: 
    prometheus: "true"
  baseURL: "https://prometheus.project.io"
  enabled: true
  retention: "30d"
  strategy:
    type: RollingUpdate
  global:
    scrape_interval: 30s
    external_labels:
      env: prod
      client: client-name
      project: project-name
      cluster: project-prod-eks

Upvotes: 0

Related Questions