CODEWITHSUNDEEP
cassandra
John
John

Reputation: 67

Why cassandra use up so much of CPU

I'm not an expert about Cassandra. Few months ago, i got an order to establish database for a social network application. For many reasons I refers from network, i choose Cassandra to store metadata of all feeds and redis to store list,zset or id (light weight and most changed data). My Cassandra cluster includes 4 nodes with replicate factor is set to 2, 4 nodes in 2 servers(each server has 2 nodes). I configured cassandra.yaml to tune Cassandra cluster as fast as possible. My system seems to be good at all till a day I review system stat. It's unbelievable. Let me explain why :

   First: I type "iostat -x 5" to view io stat. The result is as following:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.35    0.00    0.82    0.08    0.00   98.75

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    2.80     0.00    22.40     8.00     0.01    4.50   3.93   1.10
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

The CPU here is 0.35% and while we only have 4000 active users , IO is not my problem yet. The problem is here.

PID  USER      PR   NI VIRT  RES  SHR S  %CPU  %MEM  TIME+   COMMAND


4689 root      20   0 13.1g 6.8g 182m S 514.6 21.8  37374:09 java      5029 root      20   0 13.0g 6.7g 174m S  1.7 21.5  10113:45 java

The CPU cassandra node use up so much . Even a node can be up to 1400% of CPU and later it 's down to 0.5 %. I really don't know why. Who can tell me why ? and if possible, please tell me the solution to deal with it. Thanks in advance!

Upvotes: 3

Views: 6597

Answers (1)

John
John

Reputation: 67

After check debug log info, i think i caught the source of my problem. Cassandra uses up so much CPU cycles because of its GC.

Here is a log track :

INFO [ScheduledTasks:1] 2013-05-08 18:20:38,334 GCInspector.java (line 122) GC for ParNew: 269 ms for 1 collections, 3367771984 used; max is 6358564864

Each time GC runs, CPU cycle increases. In my setting, i set heap memory is equal to 6GB and i think it will lead to consume lots of CPU cycle while Cassandra does GC process. But set it to lower value can reduce performance. Who can tell me what i must do??

Upvotes: 2

Related Questions