CODEWITHSUNDEEP
kubernetesamazon-ekscockroachdbkubernetes-statefulset
etherman
etherman

Reputation: 27

How to migrate StatefulSet to different nodes?

I have a Kubernetes cluster of 3 nodes in Amazon EKS. It's running 3 pods of Cockroachdb in a StatefulSet. Now I want to use another instance type for all nodes of my cluster. So my plan was this:

  1. Add 1 new node to the cluster, increase replicas in my StatefulSet to 4 and wait for the new Cockroachdb pod to fully sync.
  2. Decommission and stop one of the old Cockroachdb nodes.
  3. Decrease replicas of the StatefulSet back to 3 to get rid of one of the old pods.
  4. Repeat steps 1-3 two more times.

Obviously, that doesn't work because StatefulSet deletes most recent pods first when scaling down, so my new pod gets deleted instead of the old one. I guess I could just create a new StatefulSet and make it use existing PVs, but that doesn't seem like the best solution for me. Is there any other way to do the migration?

Upvotes: 1

Views: 1679

Answers (1)

gohm'c
gohm'c

Reputation: 15548

You can consider make a copy of your ASG current launch template -> upgrade the instance type of the copied template -> point your ASG to use this new launch template -> perform ASG instance refresh. Cluster of 3 nodes with minimum 90% of healthy % ensure only 1 instance will be replace at a time. Affected pod on the drained node will enter pending state for 5~10 mins and redeploy on the new node. This way you do not need to scale up StatefulSet un-necessary.

Upvotes: 3

Related Questions