CODEWITHSUNDEEP
cassandracassandra-3.0
Sam
Sam

Reputation: 446

Disk space of Cassandra node is over 80%

I'm running 12 nodes of Cassandra in AWS EC2 instance, 4 of them are using almost 80% of the disk space, so compaction failed on these nodes, since the type of the server is EC2 instance, I can't add mode disk space to the existing data volume on the fly, I can't ask IT team to add more nodes to scale and spread the clustre as disk space of other nodes is less than 40%, before fixing the unbalanced cluster issue, is there any way to free up some disk space?

My question is how can I find unused sstables and move them to another partition to run compaction and free up some space?

Any other suggestion to free up some disk space. PS: I already dropped all the snapshots and backups.

Upvotes: 0

Views: 410

Answers (1)

Manish Khandelwal
Manish Khandelwal

Reputation: 2310

If you are using vnodes then data sizes difference should not be that much. Before coming to solution we must find the reason for big difference in data sizes on different nodes. You must look into logs to see if there is corruption of some big SStable which resulted in compaction failures and increase in data sizes. Or you can find something in your logs which points to the reason in increasing of disk sizes.

We faced an issue in Cassandra 2.1.16 due to some bug it happened that even after compaction old sstable files were not removed. We read the logs and identified the files which can be removed. This is an example where we found the reason of increased data size after reading the logs.

So your must identify the reason before solution. If it is a dire state you can identify keyspaces/tables which are not used during your main traffic and move those sstables in backup and remove those sstables. Once your compaction process is over you can bring them back.

Warning :Test any procedure before trying on production.

Upvotes: 1

Related Questions