CODEWITHSUNDEEP
javacassandradatastax
Gal Goffer
Gal Goffer

Reputation: 43

Unbalanced Cassandra replicas storage

In our setup we have 2 DCs, 21 Cassandra nodes in each DC, and a total of 4 replicas per record (in one of the keyspaces) - two replicas per site. Every Cassandra node is setup with 16 VNodes. We did not manually set the initial_token for each node in the cluster, rather we let Cassandra decide on the token range distribution.

My problem - when listing the token ranges in our cluster it seems that the replicas are stored in an unbalanced manner. For instance, this is one of the 672 records displayed ([42 nodes]*[16 VNodes]):

TokenRange(start_token:-8400100964809109474, end_token:-8379945255976340112, endpoints:[xxx.101.xxx.210, xxx.101.xxx.212, 10.x.108.xxx, xxx.164.xx.157], rpc_endpoints:[xxx.101.xxx.210, xxx.101.xxx.212, 10.x.108.xxx, xxx.164.xx.157], endpoint_details:[EndpointDetails(host:xxx.101.xxx.210, datacenter:ALLNTXDW, rack:RAC1), EndpointDetails(host:xxx.101.xxx.212, datacenter:ALLNTXDW, rack:RAC1), EndpointDetails(host:10.x.108.xxx, datacenter:BOTHWAKY, rack:RAC1), EndpointDetails(host:xxx.164.xx.157, datacenter:BOTHWAKY, rack:RAC1)])

Out of 672 records, every node is assigned exactly 16 times to the first term in the "endpoints" and "rpc_endpoints" tuples (as expected). But the second, third, fourth terms are populated unevenly, causing the cluster to be unbalanced. One of the nodes appears as the fourth term in 28 of the 672 tuples, whereas others appear as few as 4(!) times as the fourth term.

This leads to one of the nodes receiving over 12 percent of the data inserted to the cluster (86 out of 672 assignments in token ranges), whereas some nodes receive half of that (43 out of 672 assignments in token ranges).

Can the token range assignment for replicas be controlled? How do I approach balancing this?

Note: this cluster is production, and is already loaded with many TBs of data. In-fact this problem came to our attention as the space in one of the machines filled up to 98% whereas other machines have as little as 40 percent space used.

Thanks! Gal

Upvotes: 2

Views: 271

Answers (1)

Stefan Podkowinski
Stefan Podkowinski

Reputation: 5249

Using only 16 vnodes per host is probably not a good idea. Each vnode token will be generated randomly with the expectation that an even balancing will happen with a large enough number of tokens. The lesser number of used vnodes, the higher the variation and possibility of an uneven distribution.

Upvotes: 3

Related Questions