CODEWITHSUNDEEP
kubernetes
Felix Scheinost
Felix Scheinost

Reputation: 473

Kubernetes Graceful Node Shutdown: StatefulSet Pod stuck in completed

I have a k8s cluster with nodes which aren't special (like pets) but also not cattle in that they live a long time and I do care about them a little.

For reboots I configured kubelet to gracefully shutdown pods using shutdownGracePeriod. My thinking was, that I wanted to give e.g. databases time to flush to disk, web apps to handle in-flight requests and so on.

The problem I face after restart is that StatefulSet pods are stuck in Completed state, as the pods terminate cleanly when receiving the signal to terminate.

I have some StatefulSets with replicaCount: 1 in which case the manual solution is obvious: delete the pod and let the controller recreate it. But I wonder how that could be handled when e.g. the node that the -0 pod is on is rebooted and that pod is terminated. Would the higher-numbered pods need to be terminated as well?

Am I using StatefulSet the wrong way?

What would be the prober way to handle node reboots in an automatic way when using StatefulSet?

Upvotes: 1

Views: 540

Answers (1)

iamwillbin
iamwillbin

Reputation: 374

Yes, you are correct. If the replicaCount is 1, the manual solution is to delete the pod and let the controller recreate it. However, if the node that the -0 pod is on is rebooted and that pod is terminated, the higher-numbered pods may also need to be terminated.

Not necessarily. The challenge arises because your "non-cattle" nodes require special handling during reboots, which doesn't perfectly align with StatefulSet's standard behavior. Adapting the pod termination process or managing restarts through other means is a reasonable approach.

To achieve ordered and graceful termination of the pods in the StatefulSet, it is possible to scale the StatefulSet down to 0 prior to deletion. Implement PreStop hooks to gracefully handle cleanup or data persistence in your stateful pods. It can be used for the terminating tasks to be completed, and then a Kubernetes Job can be used to complete the tasks. It is important to handle termination gracefully so that there is minimal impact. This means saving all data that needs to be saved, closing down network connections, finishing any work that is left, and other similar tasks. Once Kubernetes has decided to terminate the pod, a series of events takes place.

See Kubernetes best practices: Terminating with Grace

Upvotes: 1

Related Questions