Cassandra - Kubernetes how to keep pod ip when restart

Question

I'm using kubernetes 1.2 example to run 2 cassandra nodes for testing. https://github.com/kubernetes/kubernetes/blob/release-1.2/examples/cassandra/README.md

I use daemonset to have one cassandra node by kubernetes node. Everything work fine till one cassandra node restart. IP address of the POD changes and nodetools status returns Node DOWN

> kubectl exec -it cassandra-lnzhj -- nodetool status fruition
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens  Owns (effective)  Host ID                                  Rack    
UN  10.216.1.4  25.22 MB   256     39.6%             786aede9-ec4f-4942-b52a-135bc3cd68ce  rack1
UN  10.216.0.3  2.11 MB    256     40.1%             457f7322-131a-4499-b677-4d50691207ba  rack1
DN  10.216.0.2  377.41 KB  256     38.8%             aa2ca115-e8ea-4c62-8d57-bfc5b3fabade  rack1

Then when i try to send a simple "select * from table;" on a keyspace with a replication factor of 2, I've this error:

Traceback (most recent call last):
File "/usr/bin/cqlsh", line 1093, in perform_simple_statement rows = self.session.execute(statement, trace=self.tracing_enabled)
File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py", line 1602, in execute result = future.result()
File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py", line 3347, in result raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'ONE'}

How to keep POD IP address in order not to have Down node when Kubernetes restart it? Is there a better way to do it with cassandra configuration?

Answer 1

Unfortunately, even PetSets or Stateful Sets as they are now called in Kubernetes 1.5 will not solve this problem. The new feature allows you to retain same hostname, but there are not guarantees that the ip address will remain the same. Quoting from their documentation:

The Pods’ ordinals, hostnames, SRV records, and A record names have not changed, but the IP addresses associated with the Pods may have changed. In the cluster used for this tutorial, they have. This is why it is important not to configure other applications to connect to Pods in a StatefulSet by IP address.

If you are not tied to kubernetes and have the flexibility of choosing other solutions, check this out.

They provide a container-based solution that combines compute, network, storage, so you have full control over all resources required by cassandra, and perform snapshot/restore, scale out, scale up/down, failover, etc. And they guarantee that the same ip address and volumes are retained for a container for its entire lifetime.

Answer 2

You can't achieve this with current Kubernetes. You need the implementation of PetSets due in v1.3.