CODEWITHSUNDEEP
cassandrambeanshintstombstone
venkat sam
venkat sam

Reputation: 168

Apache Cassandra : Hints are getting created continuosly even if all the nodes are up

I have a 5 node apache cassandra 2.0.6 cluster with 48 GB ram and 2 TB data directory and commit log directory with 93 GB capacity. The JVM heap space for cassandra is 8 GB. I use JVisualVM Mbeans plugin for monitoring the cassandra metrics. The hints are getting created continuously in all nodes even though all the nodes are up. And as hints are getting created while writing data, after sometime i am facing tombstone overwhelming exception which is aborting the queries. Could anyone please explain why is it happening and provide remedy for the same.

Upvotes: 0

Views: 1600

Answers (1)

dkblinux98
dkblinux98

Reputation: 386

The issue with hints tombstone overwhelming exception is known and there are Jiras to improve the situation.

Are you getting the tombstone ERROR or tombstone WARN in your logs? If you are hitting the tombstone ERROR then you will want to temporarily increase the threshold to avoid the error and allow your hints to process.

If your cluster continues to generate hints regularly under normal operations then it is clearly overwhelmed in some fashion and that issue needs to be addressed so that hints are not required for normal operations. The most likely cause is long GC pauses. Do you see "GC for" messages in your system logs? If so, how long on average in ms are the pauses and how frequent? How many are ParNew vs ConcurrentMarkSweep?

Upvotes: 1

Related Questions