CODEWITHSUNDEEP
kubernetesshardingkubernetes-statefulset
Jon Watte
Jon Watte

Reputation: 7228

In kubernetes, is there a way to make statefulset pods linger to finish requests on rolling update?

In Kubernetes, I have a statefulset with a number of replicas. I've set the updateStrategy to RollingUpdate. I've set podManagementPolicy to Parallel. My statefulset instances do not have a persistent volume claim -- I use the statefulset as a way to allocate ordinals 0..(N-1) to pods in a deterministic manner.

The main reason for this, is to keep availability for new requests while rolling out software updates (freshly built containers) while still allowing each container, and other services in the cluster, to "know" its ordinal.

The behavior I want, when doing a rolling update, is for the previous statefulset pods to linger while there are still long-running requests processing on them, but I want new traffic to go to the new pods in the statefulset (mapped by the ordinal) without a temporary outage.

Unfortunately, I don't see a way of doing this -- what am I missing?

Because I don't use volume claims, you might think I could use deployments instead, but I really do need each of the pods to have a deterministic ordinal, that:

The second-best option I can think of is using something like zookeeper or etcd to separately manage this property, using some of the traditional long-poll or leader-election mechanisms, but given that kubernetes already knows (or can know) about all the necessary bits, AND kubernetes service mapping knows how to steer incoming requests from old instances to new instances, that seems more redundant and complicated than necessary, so I'd like to avoid that.

Upvotes: 0

Views: 1261

Answers (2)

Jonas
Jonas

Reputation: 129065

I assume that you need this for a stateful workload, a workload that e.g. requires writes. Otherwise you can use Deployments with multiple pods online for your shards. A key feature with StatefulSet is that they provide unique stable network identities for the instances.

The behavior I want, when doing a rolling update, is for the previous statefulset pods to linger while there are still long-running requests processing on them, but I want new traffic to go to the new pods in the statefulset.

This behavior is supported by Kubernetes pods. But you also need to implement support for it in your application.

  • New traffic will not be sent to your "old" pods.
  • A SIGTERM signal will be sent to the pod - your application may want to listen to this and do some action.
  • After a configurable "termination grace period", your pod will get killed.

See Kubernetes best practices: terminating with grace for more info about pod termination.

Be aware that you should connect to services instead of directly to pods for this to work. E.g. you need to create headless services for the replicas in a StatefulSet.

If your clients are connecting to a specific headless service, e.g. N, this means that it will not be available for some times during upgrades. You need to decide if your clients should retry their connections during this time period or if they should connect to another headless service if N is not available.

If you are in a case where you need:

  • stateful workload (e.g. support for write operations)
  • want high availability for your instances

then you need a form of distributed system that does some form of replication/synchronization, e.g. using raft or a product that implements this. Such system is easiest deployed as a StatefulSet.

Upvotes: 2

Blender Fox
Blender Fox

Reputation: 5655

You may be able to do this using Container Lifecycle Hooks, specifically the preStop hook.

We use this to drain connections from our Varnish service before it terminates.

However, you would need to implement (or find) a script to do the draining.

Upvotes: 1

Related Questions