Large Write Performance Questions

Question

My company and I have purchased about 80,000$ in hardware to accomplish a goal. We have about 22,000 writes/sec in multiple application database for our Cassandra Cluster. We built 2 x nodes of Dual 3.5Ghz Xeons, 128GB RAM, Areca 1883, all top of the line high throughput. We also have a SSD RAID 10 array for Commitlog/saved_caches so that is not delayed.

The issue we have is the amount of data. In about 4 days we collected 1.8TB of data. We have no intention of ever releasing data. We then got a JBOD enclosure and put 6TB Platter drives in, 10 each, 20 total for about 110TB of space. We run fine with single replication, the issue is when we run to double replication.

We would love to add more nodes, we know that is the correct way, but at 20,000$ a node its costly. My question is, is it true to say if our write speed is the issue, that adding 10 more drives in each machine should allow for double the write speeds?

Does anyone have some of similar things going on and have some tweaks they made to Cassandra.yaml?

We did run htop for a while when we were in double replication, and CPU did seem to get a bit intensive (Read 24% average but it looks pretty close to maxed). RAM is all being used, 128GBs.

ANY thoughts on the matter will be considered and investigated.

Thanks,

Ken

Answer 1

It is not generally true that you can increase write speed simply by increasing disks, unless you are sure that you are IO bound. Cassandra batches writes (mutations go to the commitlog first, then a table in RAM, then are batch written to sstables when that table reaches a certain threshold - linear writes, so it's generally fast, even on spinning disks). At some point, you will max out the commitlog drive, fill the memtable faster than you can flush, or simply get to the point where GC can't keep up.

There are fairly large users of Cassandra who run multiple Cassandra instances on a given server simply to get the benefits of additional nodes without "just" adding disk. By running two JVMs, you can mitigate the pause times of a single node, and still take advantage of your (oversized) hardware. This is easiest if you can assign multiple IPs to your individual servers, but running on different ports also works. This is fairly atypical, and you'll need to pay close attention to your configs to avoid stepping on each other, but it will work, and will make more efficient use of your hardware than simply running huge nodes.

Answer 2

If I'm reading that correctly, you only have 2 nodes total?

If you only have 2 nodes I doubt that disk bandwidth would be the problem. Cassandra is usually CPU limited more than anything else.

Writes generally go to memory, so the disk only comes into play when memtables are flushed to disk as SStables. Now the thing that will probably kill your performance is when those SStables need to be compacted. When compaction starts happening, guess what part of the system that will stress, yup, the CPU.

You will also have a problem running repairs with huge disks like that. Usually I find that sustained transaction throughput is limited by compactions and repairs more than raw write performance.

With two nodes and single replication, you'd be splitting the load between the two nodes, with half going to one and half going to the other. If you set the replication factor to two, now every write would be going to both nodes, which is like going back in time to having a single machine database.

So I think it was a bad call to buy a small number of high end machines. You would have had much better performance with more machines where each machine was less expensive. You need more machines to spread out the load and get more CPUs into the equation.

Also you mention a disk enclosure. I hope you are not trying to use network storage with Cassandra. It needs the disks to be local.