CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-logginggoogle-cloud-monitoring
domdomegg
domdomegg

Reputation: 1816

Google Cloud Monitoring alert policy not firing when using nully labelExtractors

I'd like to get notified when there are errors in our production application. To do this, I set up Google Cloud Monitoring log-based alerts, which sends me notifications when there are errors. To be able to quickly triage these alerts, it'd be helpful to include more information. In particular, including the log message would likely be most useful.

Given that I have logs structured in different ways (both plain text and JSON logs), I set up label extractors for both values. I then put the result in the 'documentation' value so the notification includes the log message. Downloaded as JSON, my config looks like:

{
  "name": "projects/my-project-id/alertPolicies/123",
  "displayName": "Error in logs",
  "documentation": {
    "content": "${log.extracted_label.jsonMessage}${log.extracted_label.textMessage}",
    "mimeType": "text/markdown"
  },
  "userLabels": {},
  "conditions": [
    {
      "name": "projects/my-project-id/alertPolicies/123/conditions/456",
      "displayName": "Log match condition",
      "conditionMatchedLog": {
        "filter": "severity>=\"ERROR\"",
        "labelExtractors": {
          "jsonMessage": "EXTRACT(jsonPayload.message)",
          "textMessage": "EXTRACT(textPayload)"
        }
      }
    }
  ],
  "alertStrategy": {
    "notificationRateLimit": {
      "period": "300s"
    },
    "autoClose": "1800s"
  },
  "combiner": "OR",
  "enabled": true,
  "notificationChannels": [
    "projects/my-project-id/notificationChannels/789"
  ]
}

Unfortunately, when I do this the logs appear correctly associated with the alert in the Google Cloud console, but no incidents or notifications are created. I've noticed if I select just one (e.g. remove jsonMessage) then I do get notifications, but these are filtered down to only those with a textPayload.

Upvotes: 0

Views: 1114

Answers (2)

domdomegg
domdomegg

Reputation: 1816

Given that the policies can work independently, as a workaround you can create two alerting policies: one for textPayloads and one for jsonPayloads. You may also want a third policy to catch errors that don't have a jsonPayload.message or textPayload filter (using clauses in the filter key) that fall back to showing no message in the documentation.

Upvotes: 0

domdomegg
domdomegg

Reputation: 1816

The filtering behaviour described at the end of the post is not in the API docs, but appears to be only the case if the label extractor is used in the documentation.

As a workaround, you can remove the documentation block entirely. On several channels (including email and Slack), if there is no documentation, the extracted labels will still be sent through: they'll be formatted in a less accessible way, but it is better than nothing.

Upvotes: 0

Related Questions