CODEWITHSUNDEEP
javacassandra
cuilin li
cuilin li

Reputation: 1

Cassandra takes up a lot of CPU

The Cassandra cluster uses 3 nodes, among which the factor is 3. The current Cassandra version is 3.11.10，Before performing the repair operation, the CPU usage rate exceeds 1000%. After performing the repair operation, the CPU usage rate is below 100%. You must perform the repair once every hour, otherwise the CPU usage will exceed 1000%。

At present, the total keyspace in Cassandra does not exceed 6, the total write QPS is less than 1, the total read QPS is less than 2, and the total data size (all keyspaces) does not exceed 500M.

The Cassandra configuration (cassandra.yml) uses the official default configuration。

The consistency level of Cassandra is QUORUM

Below are some Cassandra statistics (via nodetool)

nodetool status: enter image description here

nodetool tpstats: enter image description here

nodetool compactstats: enter image description here

CassandraDaemon jvm options:

/usr/lib/jvm/java-8u5-sun/bin/java -Xloggc:/data/cassandra/logs/gc.log -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xms7938M -Xmx7938M -Xmn1984M -XX:+UseCondCardMark -XX:CompileCommandFile=./../conf/hotspot_compiler -javaagent:./../lib/jamm-0.3.0.jar -Dcassandra.jmx.local.port=7199 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password -Djava.library.path=./../lib/sigar-bin -XX:OnOutOfMemoryError=kill -9 %p -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/data/cassandra/logs -Dcassandra.storagedir=./../data -Dcassandra-pidfile=/data/cassandra/conf/pidfile -cp ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/HdrHistogram-2.1.9.jar:./../lib/ST4-4.0.8.jar:./../lib/airline-0.6.jar:./../lib/antlr-runtime-3.5.2.jar:./../lib/apache-cassandra-3.11.10.jar:./../lib/apache-cassandra-thrift-3.11.10.jar:./../lib/asm-5.0.4.jar:./../lib/caffeine-2.2.6.jar:./../lib/cassandra-driver-core-3.0.1-shaded.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.9.jar:./../lib/commons-lang3-3.1.jar:./../lib/commons-math3-3.2.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrent-trees-2.4.0.jar:./../lib/concurrentlinkedhashmap-lru-1.4.jar:./../lib/disruptor-3.0.1.jar:./../lib/ecj-4.4.2.jar:./../lib/guava-18.0.jar:./../lib/high-scale-lib-1.0.6.jar:./../lib/hppc-0.5.4.jar:./../lib/jackson-annotations-2.9.10.jar:./../lib/jackson-core-2.9.10.jar:./../lib/jackson-databind-2.9.10.4.jar:./../lib/jamm-0.3.0.jar:./../lib/javax.inject.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jcl-over-slf4j-1.7.7.jar:./../lib/jctools-core-1.2.1.jar:./../lib/jflex-1.6.0.jar:./../lib/jna-4.2.2.jar:./../lib/joda-time-2.4.jar:./../lib/json-simple-1.1.jar:./../lib/jstackjunit-0.0.1.jar:./../lib/libthrift-0.9.2.jar:./../lib/log4j-over-slf4j-1.7.7.jar:./../lib/logback-classic-1.1.3.jar:./../lib/logback-core-1.1.3.jar:./../lib/lz4-1.3.0.jar:./../lib/metrics-core-3.1.5.jar:./../lib/metrics-jvm-3.1.5.jar:./../lib/metrics-logback-3.1.5.jar:./../lib/netty-all-4.0.44.Final.jar:./../lib/ohc-core-0.4.4.jar:./../lib/ohc-core-j8-0.4.4.jar:./../lib/reporter-config-base-3.0.3.jar:./../lib/reporter-config3-3.0.3.jar:./../lib/sigar-1.6.4.jar:./../lib/slf4j-api-1.7.7.jar:./../lib/snakeyaml-1.11.jar:./../lib/snappy-java-1.1.1.7.jar:./../lib/snowball-stemmer-1.3.0.581.1.jar:./../lib/stream-2.5.2.jar:./../lib/thrift-server-0.3.7.jar:./../lib/jsr223//.jar: -XX:HeapDumpPath=/data/cassandra/logs/dump.log -XX:ErrorFile=/data/cassandra/logs/error.log org.apache.cassandra.service.CassandraDaemon

Upvotes: 0

Views: 1006

Answers (1)

Erick Ramirez
Erick Ramirez

Reputation: 16353

You will need to do a thread dump of the Cassandra process in order to diagnose the problem.

You need to analyse the thread dump to figure out which thread(s) is using the most CPU. Knowing the busiest thread will give you clues as to why you're seeing high CPU utilisation.

Unfortunately the data you posted isn't enough to determine what's causing the problem. Cheers!

Upvotes: 0

Related Questions