Cassandra nodes do not start

Question

I have a cassandra cluster and I tried to add and remove nodes from it in order to evaluate the throughput gain I have when resizing. The problem is that after running a add and remove node commands (also I tried cleanup with nodetool), I now can't add 2 nodes because I get this exception:

java.lang.IllegalStateException: unable to find sufficient sources for streaming range

and cassandra bin hangs.. I am not entirely sure about how cassandra works, so can anyone provide any details on it?

Thank you

edit:

Replication factor is 1 for my cluster and in total there are 10 nodes (1 seednode, 7 working nodes and 2 hanging nodes - for now)

Answer 1

If you insert data and you then proceed to remove a node you are permanently removing data fro m the cluster if your replication factor (RF) is too low. Why? Because cassandra hasn't copied the data onto any nodes (due to the low RF) and the node that was removed stored said data uniquely.

A more visual take on the problem:

RF = 1
Each node stores 25% of the data
Total dataset = [node1, node2, node3, node4]

When we remove node 2 we now have 75% of the total data because low rf + removing nodes = data loss. This source is a simple way to workout how much nodes you can loose before you don't have your full dataset.

You're going to have to remove all the data, or re-introduce the nodes that had the data. My advice is increase the replication factor.