CODEWITHSUNDEEP
cassandrareplicationdsbulk
prasanna
prasanna

Reputation: 45

Why do row counts per node differ for a 5-node cluster with a replication factor of 3?

I have 5 nodes of machines connected in a Cassandra distributed data system. I have setup the replication factor as 3.

I have understood that for a replication of 3, the data will be spread across 3 nodes based on the coordinator nodes availability. When I check for individual nodes, the row counts are differing. I have transferred some 100k of rows from csv to cassandra. Does this mean, I have to take row counts for all nodes all together to get the results ? I am using dsbulk for checking the row count.

Am I missing something here?

Upvotes: 1

Views: 146

Answers (3)

Erick Ramirez
Erick Ramirez

Reputation: 16393

The row count between nodes will never be exactly the same because of the way data is distributed around the cluster.

In a 5-node data centre, each node will roughly own 20% of the data. The keyword being "roughly" because the number of tokens owned (token ranges) by each node is not absolutely the same -- some nodes will have a slightly larger token range while some nodes have slightly less, though the differences will be tiny by percentage.

On top of that, each record is distributed randomly across nodes in the cluster using an algorithm that hashes the partition key into a token value. The random distribution of the data again introduces a level of variance so each node doesn't necessarily have exactly the same amount of data.

With just 100K partitions, the data will not get distributed equally as you would expect. It is not until you have billions of partitions will you see closer to equal distribution.

Remember that for the default Murmur3Partitioner, the number of possible hash values (tokens) for partition keys ranges from -263 to 263-1 (or roughly 2128) -- that's a very, VERY large number. By comparison, 100K is not even close to 1% of that. Cheers!

Upvotes: 2

Madhavan
Madhavan

Reputation: 649

What is your exact dsbulk count command look like? Also, what is the output of running ./dsbulk --version & via CQLSH, DESCRIBE KEYSPACE your_keyspace_name;

You would need something like below,

./dsbulk count -k keyspace_name -t table_name <other configs> --datastax-java-driver.basic.request.consistency LOCAL_QUORUM

Upvotes: 2

Andrew
Andrew

Reputation: 27314

With 5 nodes, an RF of 3, and 100k rows loaded of raw data - assuming no dropped mutations, then there is a grand total of 300k rows of data spread across the 5 nodes. (the RF of 3 x 100k).

You mention that the data is spread based on the coordinator nodes availability - but it is based on the consistent hash of the partition key of the row, as to which nodes hold the replicas.

The likelihood is that when using DSBulk you are using the default consistency level of local_one (https://docs.datastax.com/en/dsbulk/docs/reference/driver-options.html#datastaxJavaDriverBasicRequestConsistency), and that there were dropped mutations on the load. Change the consistency level to at least local_quorum / repair the cluster to bring it back to a consistent state.

Upvotes: 2

Related Questions