What do the metrics from describe on Kubernetes HorizontalPodAutoscaler on GKE?

Question

I set up my HorizontalPodAutoscaler like described here https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling to listen scale according to the number of unacked messages from my Pub/Sub. My desire is that the pods scale if there is more than 1 unacknowledged message. When I run k describe hpa I get:

Namespace:                                                                               default
Labels:                                                                                  <none>
Annotations:                                                                             kubectl.kubernetes.io/last-applied-configuration:
                                                                                           {"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"foobar-gke-prod","namespace":"defau...
CreationTimestamp:                                                                       Mon, 25 May 2020 18:01:33 -0700
Reference:                                                                               Deployment/foobar-gke-prod
Metrics:                                                                                 ( current / target )
  "pubsub.googleapis.com|subscription|num_undelivered_messages" (target average value):  200m / 1
Min replicas:                                                                            3
Max replicas:                                                                            9
Deployment pods:                                                                         5 current / 5 desired

The metrics data returned is confusing me. When I ran that command the number of unacked knowledge messages was around 4 according to the console metrics. So I don't understand what does 200m mean? Why wouldn't it say 4?

Here is my configuration for the HPA

# Template from https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: foobar-gke-prod
spec:
  minReplicas: 3
  maxReplicas: 9
  metrics:
  - external:
      metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
      metricSelector:
        matchLabels:
          resource.labels.subscription_id: prod_foobar_subscription
      targetAverageValue: "1"
    type: External
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: foobar-gke-prod

Answer 1

Reference Example:

Name:                                                                                    pubsub
...
Metrics:                                                                                 ( current / target )
"pubsub.googleapis.com|subscription|num_undelivered_messages" (target average value):  2250m / 2
Min replicas:                                                                            1
Max replicas:                                                                            4
Conditions:
Type            Status  Reason            Message
----            ------  ------            -------
AbleToScale     True    SucceededRescale  the HPA controller was able to update the target scale to 4
ScalingLimited  True    TooManyReplicas   the desired replica count is more than the maximum replica count
Events:
Type    Reason             Age   From                       Message
----    ------             ----  ----                       -------
Normal  SuccessfulRescale  7s    horizontal-pod-autoscaler  New size: 4; reason: external metric pubsub.googleapis.com|subscription|num_undelivered_messages(&LabelSelector{MatchLabels:map[string]string{resource.labels.subscription_id: echo-read,},MatchExpressions:[],}) above target

• The Metrics section gives the last value of metric observed by HPA. Fractional values are represented as milli-units. For example, in the output above there are 4 replicas of application and the current number of unacknowledged messages in Pub/Sub subscription is 9. So the average number of messages per replica is 2.25, or 2250m.

---

The metrics data returned is confusing me. When I ran that command the number of unacked knowledge messages was around 4 according to the console metrics. So I don't understand what does 200m mean? Why wouldn't it say 4?

• That means that on your case 200m/1 means that at that moment the average number of undelivered messages per replica running is 0.2(20%) at the time HPA measured.

Considerations:

• Make sure you make the readings on metrics console and HPA roughly at the same time to avoid discrepancies due to scaling running during reads.

• a reading of 4 messages for 5 pods would result in a load of 800m but at that point the hpa could be already running another scale up event.

• I encourage you to take a reading of the metrics console and hpa at the same time and verify again.

If you still think the results don't match post here with the updated hpa describe and we can take another look.

---

EDIT:

Is there anyway to make the metric not be an average across pods? I.e. if there are 5 unacked messages the metrics data would read 5000m?

From Kubernetes API Reference ExternalMetricSource v2beta1 Autoscaling:

• targetAverageValue is the target per-pod value of global metric (as a quantity).

• targetValue is the target value of the metric (as a quantity).

Note that targetAverageValue and targetValue are mutually exclusive.

So if you want the total instead of the average, just swap them on your HPA.