CODEWITHSUNDEEP
cassandradatastaxdatastax-enterprisecassandra-3.0nodetool
rajnikant7008
rajnikant7008

Reputation: 126

Cassandra node is getting stuck after nodetool decommission

I wish to decommission the Cassandra nodes as nodes need a version upgrade from 3.9 to 3.11. I log in to a particular node and execute the nodetool decommission command.

Most of the nodes are getting decommissioned from the cluster within few hours of operation but in some cases decommission is getting stuck and the node is not getting removed from the cluster, status/ state: UL. Initially, data is getting streamed across the different nodes and everything seems working fine, no error logs but after many hours it's there in the same state.

I intend to verify if decommission is successful through nodetool netstats and nodetool status whose output is below. In both the places, it says LEAVING

nodetool netstats Mode: LEAVING Not sending any streams. Read Repair Statistics: Attempted: 62537724 Mismatch (Blocking): 921768 Mismatch (Background): 746828 Pool Name Active Pending Completed Dropped Large messages n/a 0 44148788 0 Small messages n/a 1 238868718622 143 Gossip messages n/a 0 17713062 0 nodetool status Datacenter: XXXX Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UL x.x.x.x 554.42 GiB 256 20.2% b2d0fe01-a0ce-4f41-b410-0dcf4aa7f5ef rack1

Again after restarting the Cassandra service it comes in UP/NORMAL state and I repeat the same process to decommission. For some of the node, it's getting decommissioned in a single try but for others, it's taking multiple tries to getting decommissioned successfully.

Machine Information: Linux 4.4.0-101-generic x86_64 GNU/Linux
Average Load on each node : ~500GB

What should be the next step to decommission this in a more efficient way?

Upvotes: 1

Views: 3300

Answers (3)

Ranjeet Ranjan
Ranjeet Ranjan

Reputation: 960

As per Datastax Support

Use these steps to get the node to decommission.

Step 1 - Restart Cassandra/DSE on the node.

Step 2 - Run nodetool decommission again and the node should be removed from the cluster as expected.

https://support.datastax.com/s/article/Node-stuck-in-LEAVING-state-after-being-decommissioned

Upvotes: 1

Anubhuti Pandey
Anubhuti Pandey

Reputation: 1

After decommission if node is not decommissioned then run nodetool assassinate . It will forcefully remove the node .

Upvotes: 0

Shoban Sundar
Shoban Sundar

Reputation: 573

Another approach to decommission a node is to stop the node and run nodetool removenode in another live node. This will assign the token ranges to other nodes and you can simply delete the data of the removed node. You can use nodetool removenode as long as you have replication factor>1, which means your data will be available in the other nodes also. In your case if you are removing multiple nodes then it depends on your replication factor value.

Upvotes: 3

Related Questions