CODEWITHSUNDEEP
prometheusprometheus-alertmanager
PJtheOnlyDJ
PJtheOnlyDJ

Reputation: 63

Prometheus Docker fails to start with `Template: (dynamic): parse: template: :10: undefined variable "$labels"`

I am trying to run Prometheus on Nomad. Everything things look fine but when I add an alert rule with templating in annotations it fails.

here is a simple alert rule:

groups:
- name: alertmanager.rules
  rules:
  - alert: AlertmanagerDown
    expr: up{job="alertmanager"} == 0
    for: 1s
    labels:
      severity: critical
    annotations:
      description: 'Alertmanager on {{ $labels.role }} instance {{ $labels.node }} has not produced any metrics for 5 minutes'
      summary: 'Alertmanager is down on {{ $labels.role }} instance {{ $labels.node }}'

Container fails with Template: (dynamic): parse: template: :10: undefined variable "$labels"

If I remove annotations section it starts without any issue.

Upvotes: 2

Views: 1219

Answers (2)

Elaine An
Elaine An

Reputation: 11

I met the same issue when deploying Prometheus. Deploying Prometheus via a helm chart will have an issue as the same delimiters ({{ and }}) are used both by helm templating and the alerting templating in Prometheus.

For my case, the solution is:

  recordType: ALARM
  managedObjectInstance: "Namespace={{ "{{" }} $labels.app_kubernetes_io_instance }},Instance={{ "{{" }} $labels.instance }}"

More example can be found here.

Upvotes: 1

PJtheOnlyDJ
PJtheOnlyDJ

Reputation: 63

Had to set left_delimiter and right_delimiter to something other than {{ and }} https://www.nomadproject.io/docs/job-specification/template.html#left_delimiter

Upvotes: 2

Related Questions