CODEWITHSUNDEEP
cassandrajvmdatabase-tuning
rishi
rishi

Reputation: 51

Cassandra --JVM Tuning for G1GC

I am getting dropped mutation due to GC.MY young Generation GC is taking long time

G1 Young Generation GC in 1804ms. G1 Eden Space: 771751936 -> 0; G1 Old Gen: 5351931920 -> 4995416072; G1 Survivor Space: 83886080 -> 58720256;

MUTATION messages were dropped in last 5000 ms: 814 for internal timeout and 0 for cross node timeout. Mean internal dropped latency: 2874 ms and Mean cross-node dropped latency: 0 ms

In order to avoid this should I decrease the MaxGCPauseMillis option in Cassandra-env.sh.

our default value is 500ms.(JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500)

Would tuning this setting will reduce the dropped mutation messages. Do I need to increase or decrease the value.

Upvotes: 3

Views: 1790

Answers (1)

Chris Lohfink
Chris Lohfink

Reputation: 16400

1.8s GC may not be the exclusive problem if the mutation took over 5s to be processed. Is there a large amount pending in tpstats? how does cpu load look? You may be writing at a rate that Cassandra cant sustain with your hardware and data model.

You probably want to just try increasing your heap size first. Under 16gb of space is pretty hard to tune with G1. Its easier to help if you include gc logs from the time that took 1.8s, without logs its hard to tell what exactly happened. You can try decreasing gc time target, but it might not help if this long GC was an outlier. You may want to increase region size if taking too long to update remember set. Check for humongous objects and such as well which may be data modeling issue (increasing region size would help). If your getting an evacuation failure from a allocation spike, increase the heap size or set GCReservePercent to 25 instead of 10 to give it more room to handle the spikes.

Upvotes: 2

Related Questions