CODEWITHSUNDEEP
cassandrakubernetes-statefulsetkubernetes-networkpolicy
Pushpendra
Pushpendra

Reputation: 55

How do I prevent IP swaping of multiple containers running inside a pod if pod restarts/fails in kubernetes?

We have setup a Cassandra cluster in Kubernetes. We are using statefulSet & headless service. And planning to implement Multi-Rack feature.

For this, I want to deploy multiple container within a pod, and planning to add Affinity rules for that.

But one possible problem which I can think of is

In Kubernetes, IP can change during restart/failure.

So if I have 2 container within a pod, and my pod goes down. When pod comes up, there is a possibility than both container can swap their IP which will lead to inconsistent state. As primary token ranges for these 2 container are changed and other Cassandra nodes are not updated about this change.

If both container get new IPs then I don't see any problem because other Cassandra nodes will update itself. Am I right at this point ?

How to I prevent this IP swapping during restart/failure ?

Is there any document available which can help me to implement Multi-Rack feature ?

Upvotes: 1

Views: 198

Answers (1)

Manish Khandelwal
Manish Khandelwal

Reputation: 2310

So if I have 2 container within a pod, and my pod goes down. When pod comes up, there is a possibility than both container can swap their IP which will lead to inconsistent state. As primary token ranges for these 2 container are changed and other Cassandra nodes are not updated about this change.

IP swapping is not possible. Since C* cluster will complain that IP is already in use. So it is fine if C* pod comes up with some other IP other than present in C* cluster but if it tries to come up with an IP that is existing in C* cluster than pod will not be able to start and remain hung.

If both container get new IPs then I don't see any problem because other Cassandra nodes will update itself. Am I right at this point ?

Yes you are right on this.

How to I prevent this IP swapping during restart/failure ?

Instead of IP swapping we can call it IP crossover. You cannot prevent this situation. You can only take remedial action when such situation occurs. You can identify that IP crossover has occurred and then the kill the pod and see if it picks up some other IP.

Upvotes: 1

Related Questions