CODEWITHSUNDEEP
cassandracassandra-2.0nosql
alreadyexists
alreadyexists

Reputation: 355

Cassandra compaction takes too much RAM

Running an 8-node Cassandra 2.2.5 cluster with a CF about 1TB big.

Switching to LeveledCompactionStrategy for this CF causes thousands of compaction jobs, which doesn't seem to be problem by itself. But Cassandra starts using constantly increasing amount of RAM, eventually getting killed by the kernel.

What might be the reason C* uses 100G of RAM to merge some sorted files?

Upvotes: 0

Views: 1082

Answers (1)

Kurt
Kurt

Reputation: 191

The initial switch to LCS will cause a massive recompaction of all the data. If you've got a TB, that's a lot of SSTables, and a lot of compactions. When Cassandra does a compaction, it's not as simple as "merging some sorted files" as it actually has to merge updates and tombstones across the SSTables, requiring more processing than just a simple comparison.

Compactions will use RAM, however unless you've configured differently the defaults should have a limit on heap size and also on number of concurrent compactions. Having said that Cassandra will utilise cached memory as best it can, however this shouldn't cause issues.

If you have very wide partitions Cassandra will also use more off-heap memory during compactions, however 100GB is excessive. I suggest you configure your heap size, compaction throughput, and concurrent_compactors to be lower to hopefully avoid the oom-killer.

You should be switching to LCS one node/rack at a time using JMX if the cluster is under load, have a look at this guide for the info http://blog.alteroot.org/articles/2015-04-20/change-cassandra-compaction-strategy-on-production-cluster.html

If there is no load on the cluster then you could also try starting cassandra with the disable_stcs_in_l0 parameter in cassandra-env.sh to see if that helps. This will disable sizetiered compaction in L0 which should reduce the total number of compactions to recompact all the data into LCS (however the recompaction will still take a long time with 1TB of data). -Dcassandra.disable_stcs_in_l0=true

Upvotes: 2

Related Questions