CODEWITHSUNDEEP
amazon-web-servicesdockerkubernetesautoscaling
Shengxin Zhang
Shengxin Zhang

Reputation: 171

How to gracefully drain a node in EKS?

Sometimes we need to drain nodes in Kubernetes. When I manually set up a k8s cluster, I can drain the specific node then terminate that machine. While in EKS, nodes are under auto scaling group, which means I can't terminate a specific instance(node). If I manually terminate a instance, another instance(node) will be automatically added into eks cluster.

So is there any suggested method to drain a node in EKS?

Upvotes: 15

Views: 22464

Answers (3)

Alter
Alter

Reputation: 1223

For k8s < 1.23

kubectl get nodes
kubectl cordon "node-name"
kubectl drain "node-name" --ignore-daemonsets --delete-emptydir-data

For k8s >= 1.23 kubectl drain command already cordons the node.

kubectl get nodes
kubectl drain "node-name" --ignore-daemonsets --delete-emptydir-data

After draining the node, assuming you are using any Cluster Autoscaler(CA), your job is done, as the ca will detect when a node is unneeded and it will be removed.

If you do not, then you need to manually scale down the autoscaling node group of the drained node.

Upvotes: 1

Steve-Liang
Steve-Liang

Reputation: 511

These steps should work:

  1. kubectl get nodes
  2. kubectl cordon <node name>
  3. kubectl drain <node name> --ignore-daemonsets or
    kubectl drain <node name> --ignore-daemonsets --delete-emptydir-data
  4. aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <instance-id> --should-decrement-desired-capacity

For AWS autoscaling group, if you have nodes span out to multiple zones, consider delete nodes in each zones instead of all nodes from a single zone.

After the execution of the above commands, check the autoscaling group's desired number. It should decrease automatically. If you are using Terraform or another automation framework, don't forget to update your autoscaling group config in your infrastructure script.

Upvotes: 39

yasin lachini
yasin lachini

Reputation: 6026

1.) kubectl get nodes

2.) kubectl cordon <node name>

3.) kubectl drain <node name> --ignore-daemonsets --delete-emptydir-data

Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.

Upvotes: 3

Related Questions