CODEWITHSUNDEEP
cassandra
Firdousi Farozan
Firdousi Farozan

Reputation: 882

Cassandra: Data skewness

We have a 10-node Cassandra cluster. The data size looks very skewed on a few nodes. The cluster is setup using vnodes (32).

I am looking for suggestions on how to figure out why is there data skewness and how to balance the cluster.

Nodetool status output

Datacenter: us-west-2_central
=============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns    Host ID                               Rack
UN  172.20.111.20  832.35 GB  32      ?       a3e94d59-5897-46a5-afe7-b695b0179461  2b
UN  172.20.111.16  895.15 GB  32      ?       7e2f7726-3b98-4535-b44d-c522f7a0d4e6  2b
UN  172.20.110.17  1.39 TB    32      ?       925dc818-04d2-4dfa-a54b-6e0b4098fd76  2a
UN  172.20.111.18  861.17 GB  32      ?       acc23bfe-33ee-41b6-b78e-1d813697c9b1  2b
UN  172.20.110.19  835.67 GB  32      ?       5e2fd25a-5bbd-42c2-a627-a134c0e20c8b  2a
UN  172.20.110.13  932.74 GB  32      ?       d7534422-de99-49cb-a026-f3fa5ff8c16b  2a
UN  172.20.111.14  782.67 GB  32      ?       80802902-fd4a-4271-8dd9-d7421d823bd2  2b
UN  172.20.110.15  1.03 TB    32      ?       82535a06-30ee-4e7a-9e1e-ca6d33648ce8  2a
UN  172.20.110.10  697.58 GB  32      ?       bcf265b1-6e82-4d12-9e7c-2b769ccf1d3a  2a
UN  172.20.111.11  786.71 GB  32      ?       d34bcb42-f061-49d9-b4b1-41ae140d868b  2b

You can see there is a node with 1.39 TB and other node with 1.03 TB.

Upvotes: 0

Views: 894

Answers (1)

burak ibrahim sevindi
burak ibrahim sevindi

Reputation: 133

There may be different causes for this problem. For example your data model may be the reason for skewness. If you have a table that has small number of very wide partitions (millions of rows per partition) it may be the cause of this problem. These kind of partitions may end up in particular replicas, which may cause skewness.

Also, this skewness may be totally by chance. I mean, the token allocation between nodes may end up with this skewness all by chance. If this is the scenario, and if the problem gets bigger over time, you may need to replace the highly-loaded nodes with new ones.

Take a look at this article: https://www.datastax.com/dev/blog/token-allocation-algorithm

Upvotes: 1

Related Questions