CODEWITHSUNDEEP
cassandradatastaxdatastax-enterprisecassandra-2.0cassandra-3.0
Kalim
Kalim

Reputation: 319

If a node is removed in Cassandra using removenode, how to find replicas are successfully streamed

I have a 17 node Cassandra cluster. I have all keyspaces with replication factor = 3. If i remove 2 of the nodes from the cluster using nodetool removenode.

Will i have any replicas less than 3?,

Will Cassandra automatically creates another copy of replica and stream into other nodes?

Do i need to execute some commands to tell cassandra to create missing replicas and maintain replication factor=3?

How will i identify whether replicas are successfully streamed into existing nodes, any queries or nodetool commands etc

Note: Due to huge size of data and extensive use of MV's in my cluster nodetool decommission/repair and rebuild process are taking forever to execute.

Please help

Upvotes: 1

Views: 1317

Answers (2)

Aaron
Aaron

Reputation: 57748

If I remove 2 of the nodes from the cluster using nodetool removenode... Will I have any replicas less than 3?

nodetool decommission streams your data to the nodes now responsible for its replicas. nodetool removenode is only meant for situations where the node is down/unrecoverable, so that it can remove the node from gossip. As you can't stream data from a down node, no, removenode does not stream data in and of itself. It will attempt to invoke data streams to the nodes now responsible for those ranges from other replicas, but I've seen scenarios where that doesn't succeed.

So yes, if you flat-out remove two of seventeen nodes, with a RF=3 you will absolutely be down one or two replicas in some cases. Some of your queries at QUORUM will fail. If the removenode command takes a long time, you'll probably want to let it run the background to ensure it completes. If it gets stuck you can removenode force it, but then you will want to run a repair.

Will Cassandra automatically creates another copy of replica and stream into other nodes?

Again, decommission will do this, removenode will try to.

Do I need to execute some commands to tell Cassandra to create missing replicas and maintain replication factor=3?

Despite the fact that removenode initiates data streams, I'm still going to say "Yes." It's probably a good idea to run a repair on each node.

Upvotes: 1

apesa
apesa

Reputation: 12443

Nodetool removenode will stream all the data on that node out to the replicas and will maintain your RF. To find where the particular keyspace.column-family replicas are located you can use nodetool getendpoints

$ nodetool <options> getendpoints -- <keyspace> <table> key

Read more about getendpoints options here Also, removenode has a status option and a force option you can use.

$ nodetool <options> removenode -- <status> | <force> | <ID>

Read more about removenode here

Hope that helps.

Upvotes: 1

Related Questions