CODEWITHSUNDEEP
kubernetes
Lucas Fernandes
Lucas Fernandes

Reputation: 73

When using "Kubectl Drain Node" node pods, it doesn't wait for new pods to get healthy so that old ones die later

When I do an image update in my deployment, changing from version 1.0.0 to 2.0.0 for example, with these settings:

strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: platform-menu-backend

Old version 1.0.0 is ready all the time while version 2.0.0 is NOT ready. When version 2.0.0 is ready, version 1.0.0 dies, so I don't have downtime in the application.

The problem is when I use the "kubectl drain node" command. It recreates the pods that are on the drained node in another node healthy one, but it doesn't wait for the new pods to be ready and it already kills the old pod as soon as it recreates the new pod. So I have downtime in the application.

How can I make Kubernetes wait for the new pod to be healthy and only then kill the old pod?

Upvotes: 2

Views: 2879

Answers (1)

Tarum
Tarum

Reputation: 993

To avoid directly impacting your workloads when draining a node in a Kubernetes cluster, you can create a PodDisruptionBudget (PDB) for your deployment. By setting minAvailable or maxUnavailable in your PDB, the drain command will fail if it would violate these constraints. For more information, check out the Kubernetes documentation on PDBs: https://kubernetes.io/docs/tasks/run-application/configure-pdb.

Another option is to make the target node unschedulable before rolling out or restarting your deployment. This will cause your pods to be scheduled to other available nodes, allowing you to drain the original node safely. However, this is not the preferred solution for ensuring high availability of your applications.

For optimal performance, it is recommended to increase the number of nodes in your cluster and the number of replicas for your workloads. This will ensure that even if a node is drained, your application will still be up and running on another node.

Upvotes: 4

Related Questions