cassandra stateful set in kubernetes

Question

I've been trying to setup a redundant stateful set in kubernetes with the google cassandra image, as depicted in kubernetes 1.7 documentation.

According to the image used It's a stateful set with a consistency level of ONE. In my testing example I'm using a SimpleStrategy replication with a replication factor of 3, as I have setup 3 replicas in the stateful set in one datacenter only. I've defined cassandra-0,cassandra-1,cassandra-2 as seeds, so all are seeds.

I've created a keyspace and a table:

"create keyspace if not exists testing with replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }"

"create table testing.test (id uuid primary key, name text, age int, properties map<text,text>, nickames set<text>, goals_year map<int,int>, current_wages float, clubs_season tuple<text,int>);"

I am testing with inserting data from another unrelated pod, using the cqlsh binary, and I can see that data ends up in every container, so replication is successfull. nodetool status on all pods comes up with:

Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.16.0.161  71.04 KiB  32           100.0%            4ad4e1d3-f984-4f0c-a349-2008a40b7f0a  Rack1-K8Demo
UN  10.16.0.162  71.05 KiB  32           100.0%            fffca143-7ee8-4749-925d-7619f5ca0e79  Rack1-K8Demo
UN  10.16.2.24   71.03 KiB  32           100.0%            975a5394-45e4-4234-9a97-89c3b39baf3d  Rack1-K8Demo

...and all cassandra pods have the same data in the table created before:

 id                                   | age | clubs_season | current_wages | goals_year | name     | nickames | properties
--------------------------------------+-----+--------------+---------------+------------+----------+----------+--------------------------------------------------
 b6d6f230-c0f5-11e7-98e0-e9450c2870ca |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}
 5fd02b70-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}
 5da86970-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}

But then I delete one of those db replica pods(cassandra-0), a new pod springs up again as expected, a new cassandra-0 (thanks kubernetes!), and I see now that all the pods have lost one row of those 3:

 id                                   | age | clubs_season | current_wages | goals_year | name     | nickames | properties
--------------------------------------+-----+--------------+---------------+------------+----------+----------+--------------------------------------------------
 5fd02b70-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}
 5da86970-c0f8-11e7-8e29-3f611e0d5e94 |  26 |         null |          null |       null | jonathan |     null | {'goodlooking': 'yes', 'thinkshesthebest': 'no'}

...and nodetool status now comes up with:

 Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.16.0.161  71.04 KiB  32           81.7%             4ad4e1d3-f984-4f0c-a349-2008a40b7f0a  Rack1-K8Demo
UN  10.16.0.162  71.05 KiB  32           78.4%             fffca143-7ee8-4749-925d-7619f5ca0e79  Rack1-K8Demo
DN  10.16.2.24   71.03 KiB  32           70.0%             975a5394-45e4-4234-9a97-89c3b39baf3d  Rack1-K8Demo
UN  10.16.2.28   85.49 KiB  32           69.9%             3fbed771-b539-4a44-99ec-d27c3d590f18  Rack1-K8Demo

... shouldn't the cassandra ring replicate all the data into the newly created pod, and still have the 3 rows there in all cassandra pods?

... this experience is documented in github.

...has someone tried this experience, what might be wrong in this testing context?

super thanks in advance

Answer 1

I think that after bringing down the node, you need to inform the other peers from the cluster that the node is dead and needs replacing.

I would recommend some reading in order to have a correct test case.