CODEWITHSUNDEEP
cassandra
Hemalatha
Hemalatha

Reputation: 95

Cassandra Hints table for a write-heavy system can exceed 2 Bn cells limit

  1. I am using Cassandra 2.2 in which hints are stored in system.hints table. Since, the node-Id is the partition key, the table can easily exceed 2 Bn cells limit per partition in 3 hours(default hint window) for a write-heavy system. Is it preferred to reduce this window for a write heavy system?

  2. Also, for testing purposes I brought a node down and then up after a while. Ideally after hints replay, the hints table should be truncated and compacted,but I notice that that table still has all the entries, why is not cleared? Also, how to validate if replay has happened or not?

Upvotes: 0

Views: 215

Answers (1)

Chris Lohfink
Chris Lohfink

Reputation: 16400

Upgrading to 3.0 will fix a lot of the HH issues when theres a ton of them like this. Can see here for some more details, but basically using a C* table as a queue like this is kinda an anti-pattern in itself so it was changed to be more similar to the commitlog which works far better. That is absolutely your best option here.

  1. Decreasing HH window will work but be aware that you must do repairs if a node is down longer than that window since you will lose data.

  2. They are not truncated, but deleted and you need to wait for a compaction for them to be cleared. Before delivering the hints when node comes back up it should do a major compaction that would clear an up. To validate I would just read (select count(*)) from the table, any deletes would be accounted for.

ie turn off node2, insert a value and can observe:

cqlsh:keyspace1> select count(*) from system.hints where target_id = 8e821294-50de-46d4-b668-f4dce69797aa;

 count
-------
     1

(1 rows)

Start node2, and wait for HH drain :

cqlsh:keyspace1> select count(*) from system.hints where target_id = 8e821294-50de-46d4-b668-f4dce69797aa;

 count
-------
     0

Upvotes: 1

Related Questions