Performance degradation on cassandra cluster from 1 node to 4 node

Question

I am running the cassandra stress test on vm's. Below are the throughput results of cassandra cluster with defalut configuration.

Throughput

1 node : 22000 - 26000

2 node : 17000 - 20000

3 node : 16000 - 19000

4 node : 12000 - 16000

There are no any bottlenecks on CPU, network, disk, memory, JVM_Heap.

Cassandra.yaml

cluster_name: 'Test Cluster'

num_tokens: 256

seeds: "vm1"

listen_address: vm2

rpc_address: 0.0.0.0

endpoint_snitch: RackInferringSnitch

Could you please tell me the reason for degradation of the performance while adding the nodes?

Adding some more information

I reinstalled the cassandra on two vm's. Surprising results are not that bad but there are no improvements in the 2 node cluster compared to single node.

Output of the 2 node cluster

Stress on localhost

root@gsedb3:~/dsc-cassandra-2.0.9/tools/bin# ./cassandra-stress

Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")

Averages from the middle 80% of values:

interval_op_rate : 20363

interval_key_rate : 20363

latency median : 1.5

latency 95th percentile : 5.1

latency 99.9th percentile : 109.1

Total operation time : 00:00:49

END

Stress on gsedb3,gsedb4

root@gsedb3:~/dsc-cassandra-2.0.9/tools/bin# ./cassandra-stress -d gsedb3,gsedb4

Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")

Averages from the middle 80% of values:

interval_op_rate : 16420

interval_key_rate : 16420

latency median : 1.7

latency 95th percentile : 6.9

latency 99.9th percentile : 68.4

Total operation time : 00:01:03

Stress with 400 threads

root@gsedb3:~/dsc-cassandra-2.0.9/tools/bin# ./cassandra-stress -t 400

Averages from the middle 80% of values:

interval_op_rate : 24841

interval_key_rate : 24841

latency median : 4.0

latency 95th percentile : 32.2

latency 99.9th percentile : 196.5

Total operation time : 00:00:41

Stress with 400 threads on gsedb3,gsedb4

root@gsedb3:~/dsc-cassandra-2.0.9/tools/bin# ./cassandra-stress -d gsedb3,gsedb4 -t 400

Unable to create stress keyspace: Keyspace names must be case-insensitively unique >>("Keyspace1" conflicts with "Keyspace1")

Averages from the middle 80% of values:

interval_op_rate : 25273

interval_key_rate : 25273

latency median : 3.9

latency 95th percentile : 26.9

latency 99.9th percentile : 252.8

Total operation time : 00:00:40

Answer 1

You are running out of cassandra-stress not Cassandra. Typically you'll need to have more stress machines compared to the number of servers under test (2-4/1 ratio). One thing you can try to increase the amount of throughput is increasing the thread count on the stress instance.

Your goal should be to find the final capacity of the server based on CPU or IO limits. (IO will be the first to go)