Reputation: 778
I'm setting up datadog monitors/alerts and want to have alerts routed to slack or pagerduty depending on if the issue is in our production environment or not. I've created multi-alert monitors that alert correctly, but I can't figure out how to make only ones where environment.name
is equal to prod
send an alert to pagerduty, and always send them to Slack.
I was hoping to be able to do something like the following in the alert message but haven't been able to figure out a syntax that works:
[...alert message...]
{{#environment.name==prod}}@pagerduty{{/environment.name}}
@slack
For now, I've found a work around of having two monitors that are duplicates of each other where one has is scoped to production only and alerts pagerduty only and the second is for all environments and alerts slack only. However, I know this is going to become a maintenance nightmare as we grow and I'd like to know if there's a better solution.
Upvotes: 1
Views: 2642
Reputation: 2279
What you want is either the is_match
or is_exact_match
conditional variable, which are documented here (with examples).
The idea is that you can nest your messages and notifications in conditional logic arguments so that only when the monitor alerts/warns/resolves, or only when the evaluated tag scope matches certain conditions, will certain messages or notification channels be part of the alert.
So in your case you want your message to include something like this:
{{#is_exact_match "environment.name" "prod"}}
Add special prod message here
and @pagerduty or @pagerduty-foo
{{/is_exact_match}}
Add message that should always show up here
and @slack-bar
In this case, only when the "environment" tag's value is "prod" will the bracketed content be included (which includes the pagerduty notification). The non-bracketed part will always be included (which includes the slack notification).
Upvotes: 4