CODEWITHSUNDEEP
prometheusprometheus-alertmanagerpagerduty
Sujai Sivasamy
Sujai Sivasamy

Reputation: 1246

Sending custom fields in prometheus alerts to pagerduty

I have a scenario where I need to send the current time stamp from Prometheus to Pagerduty. I need to access the current time stamp from the Custom details in Pagerduty to write Event Rules based on the current time.

Currently I can see three fields inside Custom Details in Pagerduty ["firing", "num_firing", "num_resolved"]. All the alert related details are inside "firing" field but as a single string.

I thought of adding the time stamp to the Labels within the Prometheus event rules, but the problem is that I will not be able to retrieve that value from the "firing" field as it is not in a structured format.

Event details in Pagerduty received from Prometheus:

{
 "client": "AlertManager",
 "client_url": "http://********/",
 "description": "[FIRING:1] **************",
 "event_type": "trigger",
 "incident_key": "********",
 "service_key": "********",
 "details": {
  "firing": "Labels:\n - alertname = ******\n - datacenter = *****\n - instance = ******\n - instance_id = ******\n - instance_type = ******\n - job = ******\n - metrics = ******\n - node = ******\n - pod = ******\n - private_ip = ******\n - public_ip = ******\n - service = ******\n - severity = critical\nAnnotations:\n - description = ******\n - summary = ******\nSource: ******\n",
  "num_firing": "1",
  "num_resolved": "0",
  "resolved": ""
 }
}

I need to add the current time stamp in the same level as the "firing" field. Is there any way to do this via Prometheus alert manager configuration or via the Alert rules?

Upvotes: 1

Views: 3036

Answers (3)

Sutirtha Das
Sutirtha Das

Reputation: 183

You can directly add this as a custom detail to the alertmanager.yaml .

details:
  time: {{ query "time()" }}

Note that this adds the time from alertmanager's point of view so might contain a small delay.

Upvotes: 1

Chintaman
Chintaman

Reputation: 101

- alert: Alert
  for: 5m
  expr: ...
  annotations:
    timestamp: >
      time: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }} {{ end }}

I have solved same problem, as above. Cheers.!

Upvotes: 2

yamenk
yamenk

Reputation: 51876

The standard prometheus way for adding additional metadata to alerts is through annotations.

- alert: Alert
  for: 5m
  expr: ...
  annotations:
    timestamp: >
      time: {{ query "time()" }}

This should add the timestamp annotation to the alert being fired.

Upvotes: 2

Related Questions