CODEWITHSUNDEEP
kubernetesautoscalinghorizontal-pod-autoscaling
Rodrigo Boos
Rodrigo Boos

Reputation: 121

Cluster Autoscaler and Horizontal Pod Autoscaler working together

I have a cluster with Cluster Autoscaler activated and HPA for one of my deployments.

This is the HPA definition:

kind: HorizontalPodAutoscaler
metadata:
  name: hpa-resource-metrics-cpu 
spec:
  scaleTargetRef:
    apiVersion: apps/v1 
    kind: ReplicationController 
    name: hello-hpa-cpu 
  minReplicas: 1 
  maxReplicas: 10 
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50

Now in a situation where my cluster is being used very lightly, that means this deployment will only have 1 available replica.

And since the cluster is not under high usage, it could be the case that the node containing that replica is scheduled for deletion (downscaling).

In that case, it would make my deployment have a downtime (when the cluster node is deleted, the only replica for the deployment is deleted as well, so it needs to be rescheduled in a new pod). I don't want that to happen (the downtime).

From this issue: https://github.com/kubernetes/kubernetes/issues/48307, it seems that Pod Disruption Budgets are not applicable to deployments with only 1 replica.

So the only solution to my problem would be to have minReplicas set to 2?

Or is there something else I could do to prevent this downtime, and still let minReplicas as 1?

Upvotes: 0

Views: 475

Answers (1)

David Maze
David Maze

Reputation: 159781

Kubernetes has the notion of a disruption. The cluster autoscaler (or an administrator) taking a node offline is a "voluntary" disruption (as distinct from, say, the node losing power) and so you have some control over it. If you create a pod disruption budget:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: hello-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: hello

You have specified that there shouldn't be fewer than one pod, with a label app: hello, when the cluster tries to perform a voluntary disruption.

Doing this can prevent the cluster autoscaler from actually deleting the node. The examples in the PDB documentation generally have multiple replicas and can tolerate some of them being offline, so it's possible to delete 1 replica of 3 and recreate it on a different node. There is an extended example where there's not capacity in the cluster to start a rescheduled pod, and this blocks destroying a node. You might set the HPA to minReplicas: 3 to avoid this case, even if it means your system will be overprovisioned at the quietest times.

Upvotes: 1

Related Questions