How to Prometheus alerts activeAt or startAt time get in Mail notification message?

Question

I use the prometheus-server, prometheus-node-exporter , prometheus-kube-state-metrics , prometheus-alertmanager

I try to alerts mail message in the Prometheus use with "prometheus-alertmanager". But in the Alert mail message there are not define activeAt or startAt time.

When i browsing this Prometheus link and see "http://127.0.0.1:9090/api/v1/alerts" The result is :

{
  "status": "success",
  "data": {
    "alerts": [
      {
        "labels": {
          "alertname": "PodCpuUsage",
          "namespace": "default",
          "pod": "prometheus-alertmanager-0",
          "severity": "critical"
        },
        "annotations": {
          "escription": "Pod prometheus-alertmanager-0 in namespace default has CPU usage below 10000 in the last 5 minutes.\n",
          "summary": "Pod prometheus-alertmanager-0 CPU usage below threshold",
          "timestamp": "Time: 2024-08-22 08:51:46.564 +0000 UTC\n"
        },
        "state": "firing",
        "activeAt": "2024-08-22T08:31:46.565522493Z",
        "value": "3.6831480399999996e+03"
      }
    ]
  }
}

And my mail message is :

 [1] Firing
 Labels
 alertname = PodCpuUsage
 namespace = default
 pod = prometheus-alertmanager-0
 severity = critical
 Annotations
 escription = Pod prometheus-alertmanager-0 in namespace default has CPU usage below 10000 in the last 5 minutes.
 summary = Pod prometheus-alertmanager-0 CPU usage below threshold
 timestamp = Time: 2024-08-22 07:52:46.564 +0000 UTC
 Source

My alerting-rule.yaml code is :

alerting_rules.yml: |
    groups:
    - name: example_alert
      rules:
      - alert: PodCpuUsage
        expr: (sum(container_cpu_usage_seconds_total) by (pod, namespace, container) * 100) < 10000
        for: 5m
        labels:
          severity: critical
        annotations:
          timestamp: >
            Time: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}
          summary: "Pod {{ $labels.pod }} CPU usage below threshold"
          escription: |
            Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has CPU usage below 10000 in the last 5 minutes.

I want this "activeAt": "2024-08-22T08:31:46.565522493Z", time print in the mail message

But there are firing time "Time: 2024-08-22 07:52:46.564 +0000 UTC" print in my mail message.

And i try this method:

annotations:
  timestamp: >
     Time: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}

Answer 1

You can define a custom template that overrides the default used for email notifications in the prometheus alertmanager

1. In your alertmanager.yml, set the custom template path.

templates:
  - '/etc/alertmanager/email-custom.tmpl'

2. Copy over the email.tmpl.

3. Modify the email-custom.tmpl file to include the StartsAt field. The StartsAt field is the time the alert started firing.

StartsAt ≠ ActiveAt because it marks the time prometheus evaluates the pending alert to be ready to fire.

                {{ range .Alerts.Firing }}
                <tr style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">
                  <td class="content-block" style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; vertical-align: top; padding: 0 0 20px;" valign="top">
                    <strong style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">Labels</strong><br style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">


                    <div>Starts at: {{ .StartsAt }}</div> <!-- Include the StartsAt field -->


                    {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">{{ end }}
                    {{ if gt (len .Annotations) 0 }}<strong style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">Annotations</strong><br style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">{{ end }}
                    {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">{{ end }}
                    <a href="{{ .GeneratorURL }}" style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px; color: #348eda; text-decoration: underline;">Source</a><br style="margin: 0; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; box-sizing: border-box; font-size: 14px;">
                  </td>
                </tr>
                {{ end }}

4. You can test your template with the alertmanager CLI tool.

amtool template render \
--template.glob='/etc/alertmanager/mail.tmpl' \
--template.text='{{ template "email.default.html" . }}' \
--template.data='/etc/alertmanager/data-sample.json'

Here is the content of data-sample.json that you can use for testing your template.

{
  "receiver": "team-X-mails",
  "status": "active",
  "alerts": [
  {
    "annotations": {
      "summary": "Service is down"
    },
    "endsAt": "2024-08-22T11:56:32.971Z",
    "fingerprint": "848d4fa77803ce9d",
    "receivers": [
      {
        "name": "team-X-mails"
      }
    ],
    "startsAt": "2024-08-22T11:48:12.971Z",
    "status": "Firing",
    "updatedAt": "2024-08-22T11:52:32.988Z",
    "generatorURL": "http://6f0677f31ddc:9090/graph?g0.expr=up%7Bjob%3D%22services%22%7D+%3C+1\u0026g0.tab=1",
    "labels": {
      "alertname": "InstanceDown",
      "instance": "idonotexists:564",
      "job": "services"
    }
  },
  {
    "annotations": {
      "summary": "Service is down"
    },
    "endsAt": "2024-08-22T11:56:32.971Z",
    "fingerprint": "e4a67535b46abfe9",
    "receivers": [
      {
        "name": "team-X-mails"
      }
    ],
    "startsAt": "2024-08-22T11:48:12.971Z",
    "status": "firing",
    "updatedAt": "2024-08-22T11:52:32.988Z",
    "generatorURL": "http://6f0677f31ddc:9090/graph?g0.expr=up%7Bjob%3D%22services%22%7D+%3C+1\u0026g0.tab=1",
    "labels": {
      "alertname": "InstanceDown",
      "instance": "prometheus:9090",
      "job": "services"
    }
  }

  ],
  "groupLabels": {},
  "commonLabels": {},
  "commonAnnotations": {},
  "externalURL": ""
}