DSE Garbage Collection on non-heap CMS Perm Gen took a long time

Question

I experience prolonged GC pause (> 10 seconds) in Datastax-enterprise's Cassandra/Solr package. After monitoring for days I found it only happens when GC of CMS Perm Gen happens as shown in the chart. And the long GC happens at EVERY flection point of the chart, when PermGen GC happens. And every time when Perm Gen GC kicks in, there's a long pause that cause client sessions timeout!

https://www.dropbox.com/s/qgdcurprvc1sees/permgen_gc.png

Heap GC is normal and has no pauses, only and always get the long pause in the Non-heap Perm Gen GC, which always happens when the server is at non-peak hours.

![enter image description here][1]

JVM options used by DSE:

-ea -javaagent:/usr/local/dse/resources/cassandra/lib/jamm-0.2.5.jar 
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
-Xms16384M -Xmx16384M -Xmn5461M -XX:+HeapDumpOnOutOfMemoryError 
-Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-Djava.net.preferIPv4Stack=true -Dcassandra.load_ring_state=false 
-Dcassandra-foreground=yes -Dsearch-service=true
-Dtomcat.logs=/var/log/dse/tomcat -DName=SI2_DSE
-Ddse.solr.data.dir=/data/solrIndexRamDisk
-Djava.library.path=/usr/local/dse/resources/hadoop/native/Linux-amd64-64/lib

JVM Info

• Virtual Machine: Java HotSpot(TM) 64-Bit Server VM version 20.12-b01
• Vendor: Sun Microsystems Inc.
• JIT compiler: HotSpot 64-Bit Tiered Compilers

Heap Info

• Current heap size: 10,247,153 kbytes
• Maximum heap size: 16,218,048 kbytes
• Committed memory: 16,218,048 kbytes
• Pending finalization: {0} objects

VM Server Info

• Operating System: Linux 2.6.32-279.1.1.el6.x86_64
• Architecture: amd64
• Number of processors: 32
• Committed virtual memory: 39,845,596 kbytes
• Total physical memory: 99,018,824 kbytes
• Free physical memory: 58,184,572 kbytes
• Total swap space: 4,194,296 kbytes
• Free swap space: 4,194,296 kbytes

Answer 1

If you can use straight Solr, you could try Heliosearch, which tries to tackle the problem of GC pauses with off-heap data.

http://heliosearch.org/off-heap-filters/

Answer 2

Use -XX:+CMSClassUnloadingEnabled, it allows the CMS collector to sweep the permgen during oldgen GC and unload classes that are no longer in use. Link : http://blog.redfin.com/devblog/2012/06/cmsclassunloadingenabled-at-redfin.html#.UwWeO4XqPK0

Use 100 or 200 MB size for perm generation instead of 60 MB.

try it out and share if it solves your problem