CODEWITHSUNDEEP
cassandra
juan
juan

Reputation: 81962

How to force Cassandra not to use the same node for replication in a schema with vnodes

Installing Cassandra in a single node to run some tests, we noticed that we were using a RF of 3 and everything was working correctly.

This is of course because that node has 256 vnodes (by default) so the same data can be replicated in the same node in different vnodes.

This is worrying because if one node were to fail, you'd lose all your data even though you thought the data was replicated in different nodes.

How can I be sure that in a standard installation (with a ring with several nodes) the same data will not be replicated in the same "physical" node? Is there a setting to avoid Cassandra from using the same node for replicating data?

Upvotes: 1

Views: 622

Answers (3)

Brad Schoening
Brad Schoening

Reputation: 1381

Cassandra stores replicas on different nodes in the same keyspace. It would be nonsensical to have multiple replicas in the same keyspace. If the replication factor exceeds the number of nodes, than the number of nodes is your replication factor.

But, why is this not an error? Well, this allows for provisioning more nodes later.

As a general rule, the replication factor should not exceed the number of nodes in the cluster. However, you can increase the replication factor and then add the desired number of nodes later.

Upvotes: 0

Eugen Constantin Dinca
Eugen Constantin Dinca

Reputation: 9150

Without vnodes each physical node owns a single token range. With vnodes each physical node will own multiple, non-consecutive token ranges (aka a vnode), and furthermore vnodes are randomly assigned to physical nodes.

Which means that even when data gets replicated on the vnodes right next to the primary replica's node (i.e. when using SimpleStrategy) the replicas will - with high probability but not guaranteed - be on different physical nodes.

This random assignment can be seen in the output of nodetool ring.

More info can be found here.

Upvotes: 0

xmas79
xmas79

Reputation: 5180

Replication strategy is schema dependent. You probably used the SimpleStrategy with RF=3 in your schema. That means that each piece of data will be placed on the node determined by the partition key, and successive replicas will be placed on the successive nodes. In your case, the successive node is the same physical node, hence you get 3 copies of your data there.

Increasing the number of nodes solves your problem. In general, your data will be placed in different physical nodes when your replication factor RF is less than/equal to your number of nodes N.

The other solution is to switch replication strategy and use the NetworkTopologyStrategy, usually used in multi datacenter clusters, and where you can specify how many replicas you want in each data center. This strategy

places replicas in the same data center by walking the ring clockwise until reaching the first node in another rack. NetworkTopologyStrategy attempts to place replicas on distinct racks because nodes in the same rack (or similar physical grouping) often fail at the same time due to power, cooling, or network issues.

Look at DataStax documentation for more information.

Upvotes: 2

Related Questions