CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-monitoring
omricoco
omricoco

Reputation: 921

GCP Monitoring - Incident does not contain system labels

I've created an alert policy via the GCP console.This policy sends incidents to a PubSub notification channel.

For example: a high CPU utilization policy for containers

{
  "name": "...",
  "displayName": "...",
  "documentation": {},
  "conditions": [
    {
      "name": "...",
      "displayName": "Kubernetes Container - CPU usage time",
      "conditionThreshold": {
        "aggregations": [
          {
            "alignmentPeriod": "300s",
            "perSeriesAligner": "ALIGN_RATE"
          }
        ],
        "comparison": "COMPARISON_GT",
        "duration": "0s",
        "filter": "metric.type=\"kubernetes.io/container/cpu/core_usage_time\" resource.type=\"k8s_container\"",
        "thresholdValue": 0.04,
        "trigger": {
          "count": 1
        }
      }
    }
  ],
  "alertStrategy": {
    "autoClose": "604800s",
    "notificationPrompts": [
      "OPENED"
    ]
  },
  "combiner": "OR",
  "enabled": true,
  "notificationChannels": [
    "..."
  ],
  "creationRecord": {
    "mutateTime": "...",
    "mutatedBy": "..."
  },
  "mutationRecord": {
    "mutateTime": "...",
    "mutatedBy": "..."
  }
}

Once I trigger this alert, and get the incident on the PubSub side, the metadata field for system_labels is always empty:

    "metadata": {
      "system_labels": {},
      "user_labels": {}
    },

Even though, if I use the metrics explorer to view this resource I see that these labels are populated.

Any suggestions?

Upvotes: 2

Views: 1507

Answers (2)

Chandra Kiran Pasumarti
Chandra Kiran Pasumarti

Reputation: 702

It's not a bug. Values for metadata.* variables are available only if the labels are explicitly included in a condition's filter or grouping for cross-series aggregation. That is, you must refer to the metadata label in either filter or grouping for it to have a value for the template. For more information refer to the documentation.

Upvotes: 3

Ismael Clemente Aguirre
Ismael Clemente Aguirre

Reputation: 729

According with this documentation when you create a channel, you need to:

  • Enable the Pub/Sub API and create a topic.
  • Configure the notification channel for a topic.
  • Authorize a service account.
  • Set the notification channel in an alerting policy.

To use a Pub/Sub notification channel in an alerting policy, select Pub/Sub as the channel type, and then select the topic. By default, the alerting backend attempts to generate a JSON packet with version 1.2 formatting.

By default the metadata field needs to have the following values:

"metadata": {
  "system_labels": { "labelkey": "labelvalue" },
  "user_labels": { "labelkey": "labelvalue" }
},

If the “system_labels” field does not present the “labelkey" and “labelvalue”, you need to review your configuration with the service account and be sure that you set the notification channel in the alerting policy correctly.

Please review the examples of JSON packet and the schema.

In the same article you will find detailed instructions about Create a Channel on demand and editing or deleting channels.

You will need to enable a filter in your policy, in order to see the fields under “metadata”:{ populated (whatever field you want to).

In order to confirm that, I replicated your scenario in Google Cloud. I created an alert and configured pub/Sub as one of the notification channels.

And, I proceeded to stress the VM to trigger the alerts. When I used a filter in the policy (In this case, the “system_labels=name”), the “metadata” field in the message’s topic in Pub/Sub showed me:

"metadata": {
      "system_labels": {"name": "pubsubtest-1"},
      "user_labels": {}
    },

But when I did not configure, the filter in the policy showed me:

"metadata": {
      "system_labels": {},
      "user_labels": {}
    },

Upvotes: 0

Related Questions