CODEWITHSUNDEEP
kuberneteskubernetes-statefulset
red888
red888

Reputation: 31662

Why isn't Kubernetes deleting a Stateful Set's pod during a rolling deployment?

I'm using the default rolling deployment strategy. I don't explicitly set the updateStrategy and when I check the Stateful Set resource I see it's using rollingUpdate:

  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate

It is configured with 3 replicas.

The initial deployment of the Stateful Set failed. The first replica is failing ready probes.

I redeployed with a config fix and also modified an environment variable on the Stateful Set.

If this was a Deployment k8s would have created a new revision and new pods for that revision, then after those pods were healthy it would have deleted the old revisions pods.

I was expected the failing replica to be deleted and recreated when I redeployed, but k8s does nothing. It does not delete or restart the broken pod and the deployment is hung up.

I can manually delete the pod, but is that expected behavior? Will every rolling deployment of a Stateful Set require manual intervention?

Edit

As I said, the first deploy of this Stateful Set failed. Does that mean I am currently in this state? https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#forced-rollback

Is doing what I want as easy as changing the Pod Management Policy to Parallel?

Upvotes: 0

Views: 1519

Answers (1)

glv
glv

Reputation: 1187

The Pods in the StatefulSet are updated in reverse ordinal order. The StatefulSet controller terminates each Pod, and waits for it to transition to Running and Ready prior to updating the next Pod. Note that, even though the StatefulSet controller will not proceed to update the next Pod until its ordinal successor is Running and Ready, it will restore any Pod that fails during the update to its current version. Pods that have already received the update will be restored to the updated version, and Pods that have not yet received the update will be restored to the previous version. In this way, the controller attempts to continue to keep the application healthy and the update consistent in the presence of intermittent failures.

https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/

In summary, even though you applied a new version of your StatefulSet, it was actually "working" to restore the existing version and restore service.

Try applying a new version of your working StatefulSet and you will see that the rolling update will work fine.

Upvotes: 0

Related Questions