CODEWITHSUNDEEP
cassandra
Chart
Chart

Reputation: 53

When adding nodes to cassandra, wait for compactions?

I have a 6 node cluster that is getting cramped in capacity, so I are added 6 new nodes to help. I added our first node today, it went from UJ to UN without issues, and then it started to work on a lot of compaction jobs. This process is currently ongoing and taking quite a bit of time, and the question I have is if there is a drawback to not letting the compactions stabilize to a normal level before adding the next node to the cluster.

I assume it will make a difference in the amount of data that "new node" will stream over to "new node 2" when the token re-partitioning happens, but how negative would that impact be I am not certain.

Should I wait? or can I add the next node right away?

Upvotes: 1

Views: 187

Answers (1)

TomerSan
TomerSan

Reputation: 1578

(Disclaimer: I'm a ScyllaDB employee)

When out-scaling your Cassandra cluster, once your 1st new node moved to UN (Up Normal) state it means that streaming has ended. There's no need to wait for compactions to end before adding the 2nd new node.

Aside from performance issues, due to the multiple parallel compactions in the background, there should not be a problem or issues.

I would also recommend running nodetool cleanup command on all the nodes in the cluster except the new node/s that has just been added. It will remove keys that no longer belong to the node. You should Run this command one node at a time. This can be postponed for low demand hours (assuming a production system).

Upvotes: 3

Related Questions