Cluster Autoscaler not scaling up with nodeSelector

Question

I have a Kubernetes cluster running various apps with different machine types (ie. cpu-heavy, gpu, ram-heavy) and installed cluster-autoscaler (CA) to manage the Auto Scaling Groups (ASG) using auto-discovery.

Kubernetes version: EKS 1.11

Cluster Autoscaler: v1.13.2

I have configured my ASGs such that they contain the appropriate CA tags. These ASGs are usually scaled down to 0 nodes, and will be scaled up according to workload. Going into the CA logs, I have also verified that the CA acknowledges the existence of the ASGs. However, whenever I try to create pods with nodeSelectors, the CAs doesn't scale up the corresponding ASG and responds with "predicate failed: nodeSelector(s) did not match". I have added the appropriate node labels to the ASG's tags as well.

Could it be a limitation due to scaling up from 0 nodes?

Can't seem to find similar problems online. Any feedback would help! Thanks!

Answer 1

I'm not sure which cloud provider you use.

It seems CA is unable to scale up with node labels when using nodeSelector, However if you change the way you define the required label and use nodeAdffinitiy it will scale up.

I talked to Amazon support and they claim it's because the CA is unable to read the userdata script and read the labels.

In any case this seems to be only required to scale up from 0 nodes

Also it is stated here https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#scaling-a-node-group-to-0 that you should tag the ASG and it should work

Answer 2

Update: CA Documentation update

I had the incorrect ASG tag. Ensure that you have k8s.io/cluster-autoscaler/node-template/label/some-label to ensure that CA can see your node labels when the ASG is scaled down to 0. Currently, CA doesn't create node labels based on ASG tags, but this feature is in the pipeline.

Answer 3

I have a much simpler setup here. I have just one group of ASG. The minimum amount is 1 and 10 maximum.

Maybe auto discovery is not working the way it should. It is the best approach, but it is not working. I would give a try on the example below to manage multiple ASG.

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-multi-asg.yaml

I have the same doubt as you, I am not sure if autoscaler works properly ranging from 0 nodes.