CODEWITHSUNDEEP
kubernetesgoogle-kubernetes-enginehpa
GavinE-OVO
GavinE-OVO

Reputation: 271

Horizontal Pod Autoscaling via custom-metrics-stackdriver-adapter fails for metrics "more than 63 characters"

I have a Kubernetes cluster running in Google Kubernetes Engine where I wish to scale a deployment based on the number of messages a GCP Pub/Sub subscription has outstanding. I found and followed a guide from Google covering exactly this scenario which involves deploying the custom-metrics-stackdriver-adapter.

However, because my subscriptions name is over 63 characters in length when I run kubectl describe hpa <MY_HPA_NAME> I get the following events.

Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetExternalMetric       12s (x2 over 27s)  horizontal-pod-autoscaler  invalid label value: "<MY_VERBOSE_SUBSCRIPTION_NAME>": must be no more than 63 characters
  Warning  FailedComputeMetricsReplicas  12s (x2 over 27s)  horizontal-pod-autoscaler  failed to get pubsub.googleapis.com|subscription|num_undelivered_messages external metric: invalid label value: "<MY_VERBOSE_SUBSCRIPTION_NAME>": must be no more than 63 characters

How do I go about using a Subscription name which is over 63 characters as a metric to trigger HPA?

Extra questions

I did try looking into what metric values where being exposed but when I queried Kubernetes Custom Metrics API I can't see any actual values for that metric by doing the following:

$ kubectl get --raw "http://localhost:8001/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/pods/*/pubsub.googleapis.com|subscription|num_undelivered_messages" | jq .

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/%2A/pods/%2A/pubsub.googleapis.com%7Csubscription%7Cnum_undelivered_messages"
  },
  "items": []
}

However, as you can see this returned an empty "items" lists suggesting there were no values recorded. I assumed that this was due to all of my subscription names being over 63 characters in length. To test this I made a new subscription with a very short name and recreated the HPA to be triggered by this new subscription. This time the pods actually scaled how I expected but when I queried the Custom Metric API as above it still returned and empty "items" list.

Should I expect to see values in that response (and their labels)? How was Kubernetes able to scale the pods if no "items" were present?

Upvotes: 0

Views: 1118

Answers (1)

Arghya Sadhu
Arghya Sadhu

Reputation: 44657

Here is design documentation of the kubernetes. There is no way to avoid the restriction other than shortening the label name.

rfc1035/rfc1123 label (DNS_LABEL): An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters, with the '-' character allowed anywhere except the first or last character, suitable for use as a hostname or segment in a domain name.

Check the External Metrics API instead of Custom Metrics API for metrics items.

$ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq

https://itnext.io/google-kubernetes-engine-horizontalpodautoscaler-with-external-metrics-from-pubsub-28780c300305

Upvotes: 1

Related Questions