CODEWITHSUNDEEP
cassandracassandra-3.0
DollyShukla
DollyShukla

Reputation: 107

Cassandra node decommission stuck in UL

Cassandra decommission is in UL state , today is 5th day of process. Streams completed within 3 days only, since then state is UL .

Cluster details : 5 nodes , RF 2 , all in same DC.

Analysis :

  1. No streams in netstats command.
  2. Data size of remaining nodes has increased.
  3. I can see writing logs coming in the decommissioned node log file.
  4. No error is there on any node of cluster.

Please comment on :

  1. Is this decommission process completed, how to know more about it ?
  2. If it is stuck , restarting a node will not help , because it will start from fresh.
  3. Any specific property required to be added ?
  4. Any specific line in log which can tell about decommissioning process?

Upvotes: 1

Views: 548

Answers (1)

Aaron
Aaron

Reputation: 57808

I have seen the decommission process get stuck sometimes. Here are the steps that I'd recommend:

  1. Stop the node and remove it from the cluster. This operation will also stream data in an attempt to prevent data loss.
  2. Run a repair (on each node), and then check for data consistency.
  3. If you notice data is missing, re-add the node, and retry the decommission process.

Upvotes: 0

Related Questions