CODEWITHSUNDEEP
apache-sparksolrcassandradatastax
peter
peter

Reputation: 684

I don't know why my cassandra has such a large hints table?

My cluster has 2 datacenters. The first datacenter has 1 node with spark and solr enabled, the other datacenter has 5 nodes with spark enabled. After a while all the nodes have a couple of 100GB hints table. But my node status is all available and normal. It doesn't show the nodes are down but the hints table just keeps on increasing. The consequence is that the after a while all the disc space will be used and the nodes die and the system crashes. I don't know why this happens?

Thanks for your suggestions.

Upvotes: 2

Views: 1826

Answers (1)

Ben Slater
Ben Slater

Reputation: 126

You don't say which nodes/data center have the growing hints tables but my guess would be that the 1 node data center does not have sufficient processing capacity to keep up with the writes replicated from the 5 nodes data center. I would say that this is particularly the case when using Spark - Spark writes can easily overwhelm a cluster unless you're using the cassandra.output parameters to control write speed.

Another factor here is the consistency level you are using - a LOCAL_* consistency level will only require writes to be written to the local DC for the operation to be considered a success (and hints will be stored for replication to the other DC).

Cheers Ben

Upvotes: 2

Related Questions