CODEWITHSUNDEEP
timealertprometheus
Eric C
Eric C

Reputation: 41

Populating a Date/Time in Prometheus Alert Manager E-mail

So i have an alert rule that gets fired in prometheus when a queue length has been long for a certain period of time.

Through the alert manger, I am able to create and receive e-mails.

My question now is, as part of my e-mail body, I want to have the Date and Time that either the alert manager triggered the e-mail, or of when the alert was fired.

I am unsure how to do this. Whether I can create a label in the alert and populate it somehow with the current date/time, or what? Any ideas?

Upvotes: 2

Views: 13636

Answers (3)

Tuxdude
Tuxdude

Reputation: 49583

Although the answer from Chintaman provides the human readable timestamp, the timestamp unfortunately is always in UTC.

This might be the desired behavior for some users, but not all (if you are like me). For my scenario, I find it convenient to have a timestamp annotation which is human readable and in the same time zone as I am, especially when all the oncallers are also in the same time zone.

For such cases, the following solution works better. Prometheus doesn't provide any functions which can readily do this. However, we can leverage the toTime function provided by prometheus to convert the result to a *time.Time object, get the local version of the timestamp from this object, and then run it through the go library provided .Format() function to format it in the format we want.

The below example results in a timestamp of this fashion:

Sat Aug 17 2024 12:51:12.476 PDT UTC-07:00

- alert: InstanceDown
  expr: up == 0
  for: 5m
  annotations:
    title: Instance {{ $labels.instance }} down
    summary: Instance {{ $labels.instance }} of job {{ $labels.job }} is down
    timestamp: >-
        {{ with query "time()" }}{{ $t := . | first | value | toTime }}{{ $t.Local.Format "Mon Jan 2 2006 15:04:05.000 MST UTC-07:00" }}{{ end }}

Upvotes: 0

Chintaman
Chintaman

Reputation: 101

- alert: Alert
  for: 5m
  expr: ...
  annotations:
    timestamp: >
      time: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}

I still find iterating alerts and getting the value of timeseries or timestamp in alert text as difficult. So I have solved this problem in above way. It works, and I am able to get timestamp / timeseries of alert in email body. Cheers.!

Upvotes: 6

brian-brazil
brian-brazil

Reputation: 34172

The alerts in the Alertmanager templates have a StartsAt attribute you could use.

Upvotes: 2

Related Questions