CODEWITHSUNDEEP
kubernetesgoogle-cloud-platformgoogle-kubernetes-enginehpa
CuriousPaul
CuriousPaul

Reputation: 46

Kubernetes HPA not working: unable to get metrics

My pod scaler fails to deploy, and keeps giving an error of FailedGetResourceMetric:

Warning  FailedComputeMetricsReplicas  6s    horizontal-pod-autoscaler  failed to compute desired number of replicas based on listed metrics for Deployment/default/bot-deployment: invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

I have ensured to install metrics-server as you can see when I run the the following command to show the metrics-server resource on the cluster:

kubectl get deployment metrics-server -n kube-system

It shows this: metrics-server

I also set the --kubelet-insecure-tls and --kubelet-preferred-address-types=InternalIP options in the args section of the metrics-server manifest file.

This is what my deployment manifest looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bot-deployment
  labels:
    app: bot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bot
  template:
    metadata:
      labels:
        app: bot
    spec:
      containers:
      - name: bot-api
        image: gcr.io/<repo>
        ports:
        - containerPort: 5600
        volumeMounts:
        - name: bot-volume
          mountPath: /core
      - name: wallet
        image: gcr.io/<repo>
        ports:
        - containerPort: 5000
        resources:
          requests:
            cpu: 800m
          limits:
            cpu: 1500m
        volumeMounts:
        - name: bot-volume
          mountPath: /wallet_
      volumes:
      - name: bot-volume
        emptyDir: {}

The specifications for my pod scaler is shown below too:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: bot-scaler
spec:
  metrics:
    - resource:
        name: cpu
        target:
          averageUtilization: 85
          type: Utilization
      type: Resource
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bot-deployment
  minReplicas: 1
  maxReplicas: 10

Because of this the TARGET options always remains as /80%. Upon introspection, the HPA makes that same complaint over and over again, I have tried all options, that I have seen on some other questions, but none of them seem to work. I have also tried uninstalling and reinstalling the metrics-server many times, but it doesn't work.

One thing I notice though, is that the metrics-server seems to shut down after I deploy the HPA manifest, and it fails to start. When i check the state of the metrics-server the READY option shows 0/1 even though it was initially 1/1. What could be wrong?

I will gladly provide as much info as needed. Thank you!

Upvotes: 0

Views: 1732

Answers (1)

ir0h
ir0h

Reputation: 361

Looks like your bot-api is missing it's resource request and limit. your wallet has them though. the hpa uses all the resources in the pod to calculate the utilization

Upvotes: 1

Related Questions