CODEWITHSUNDEEP
cassandra
kingluo
kingluo

Reputation: 1795

Cassandra: Does node add/remove requires all cluster nodes be present?

When a new node added, due to the even distribution of the new tokens, the current nodes of the ring should migrate data to this new node.

So, does it requires all nodes be present? If not, then if some nodes are down, then it will miss the data migration of those parts, how and when to fix it? When those nodes come back?

Similarly, the node removal would migrate its data to other nodes, so it seems that all other nodes must be present, otherwise it would lost data?

Upvotes: 1

Views: 453

Answers (1)

Simon Fontana Oscarsson
Simon Fontana Oscarsson

Reputation: 2134

Cassandra requires all nodes to be present. I tried adding a node when another node was down and received this error:

Exception (java.lang.RuntimeException) encountered during startup: A node required to move the data consistently is down (/172.17.0.2). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false
java.lang.RuntimeException: A node required to move the data consistently is down (/172.17.0.2). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false
    at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:275)
    at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:158)
    at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
    at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1212)
    at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:891)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:657)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:570)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
ERROR 09:53:37 Exception encountered during startup

Same happens when I try to add node in different dc.

As for removing a node. Decomission only works if all replicas are up. Decomission will start by streaming all data to new replicas, if streaming fails then decomission fails.

$ nodetool decommission
error: Stream failed
-- StackTrace --
org.apache.cassandra.streaming.StreamException: Stream failed
    ...

You can still force a remove by shutting the node down and doing nodetool removenode. This will not stream data but will remove node from cluster. Use with caution.

There is also nodetool assasinate if neither decomission or removenode works.

Upvotes: 2

Related Questions