Reputation: 1
I’ve created a log-based alerting policy in GCP that sends notifications to a Pub/Sub topic when an incident is triggered. The Pub/Sub messages are processed by a Cloud Run function, which sends them to the Slack. Everything works as expected when the incident is first triggered or if the logs continue to match the policy for same incident. However, the issue arises after I acknowledge the incident in the GCP UI, I still continue to receive notifications, and the Pub/Sub messages still shows the incident 'state': 'open'
instead of "acknowledged."
Config example:
{
"displayName": "CloudSchedulerJob_ERROR+_Test",
"conditions": [
{
"displayName": "Log match condition",
"conditionMatchedLog": {
"filter": "severity>=ERROR\nresource.type=\"cloud_scheduler_job\"\njsonPayload:*\nresource.labels.job_id:*",
"labelExtractors": {
"debugInfo": "EXTRACT(jsonPayload.debugInfo)",
"job_id": "EXTRACT(resource.labels.job_id)"
}
}
}
],
"alertStrategy": {
"notificationRateLimit": {
"period": "7200s"
},
"autoClose": "604800s"
},
"notificationChannels": [
"projects/xxx/notificationChannels/10557114610890226899"
],
"severity": "ERROR"
}
Once an incident is acknowledged, I expect no further notifications for that incident unless there's a state change (e.g., it gets resolved and new logs reoccurs). Or, if notifications do continue, I would expect the incident state to be updated to "acknowledged" in the CloudEvent message.
I have checked documentation and issue tracker for relevant information but couldn’t find any specific details addressing this problem. I confirmed that there are no additional logs creating new incidents. Reviewed the alert policy configuration and searched for any settings related to acknowledgment handling but not found such settings that control notification behavior for acknowledged incidents.
Thank you in advance for any insights or suggestions you might have!
Upvotes: 0
Views: 33