CODEWITHSUNDEEP
javacassandra
AlexanderW
AlexanderW

Reputation: 123

Cassandra Nodes dying frequently with Heap Space Error

I have a problem with Cassandra Nodes dying regulary with 'java.lang.OutOfMemoryError: Java heap space' exceptions.

My Setup consists of 5 Cassandra 2.0.11 Nodes running on 5 VM's. Each VM has 8GB RAM, 100GB disk capacity and a reasonably fast CPU.

I already experimented with increasing the heap-size. Currently it is set to the default value (1/4th of 8GB=2GB).

Memory is filled really fast and seams to be the limiting factor. How can I force cassandra to use less memory? I can tolerate slower write-operations in exchange for stability.

Currently I only write with no updates, reads or deletes. I write time series with ~100000 values per file. Concurrency-level is QUORUM, replication-factor is 3. I use the java-driver from datastax.

Tables are created like this:

"CREATE TABLE IF NOT EXISTS %s.%s(\n" +
        "ts_type text,\n" +
        "ts_name text,\n" +
        "year int,\n" +
        "time timestamp,\n" +
        "value double,\n" +
        "PRIMARY KEY((ts_type, ts_name), year, time));"

Data is written like this:

for (final Double value : data) {
    final Insert insertStatement = (Insert) QueryBuilder.insertInto(keyspace, tableName)
            .value("ts_type", tsType)
            .value("ts_name", tsName)
            .value("time", timestampAsDate)
            .value("year", timestamp.getYear())
            .value("value", value)
            .setConsistencyLevel(consistencyLevel);
    batch.add(insertStatement);
    zeitpunkt = zeitpunkt.plus(period);
    if (index++ % 200 == 0) {
        sets.add(client.executeAsync(batch));
        batch = (Batch) QueryBuilder.unloggedBatch().setConsistencyLevel(consistencyLevel);
    }
}

Here is a stacktrace of a dying node: http://pastebin.com/tTNRgJMP

As you see, GC took a really long time here.

Here is a heap dump of a dying node: https://i.sstatic.net/jEUrI.jpg

Any idea what I am doing wrong?

Thanks in advance for your help.

Upvotes: 2

Views: 1602

Answers (2)

Nick
Nick

Reputation: 1

I just got done wrestling with Cassandra (2.0) over heap space issues. I had been running 3 VM nodes, 8GB RAM each, replication 1. Needless to say, not optimal.

Here's what I use it for and have found: I am storing a very long multipart key ((uuid), text, text, text, int) to reference a value (text) and a couple of other bits of tracking information, that are really not necessary, but nice to have, that take the form of two more ints. I also had (past tense) an index on one of those extra nice to have fields. Cassandra complained, quite often, that it was taking way too much time to process my batch inserts, about 4000 of them all at once each minute. And it would usually crash with a heap space error if/when I tried to do a nodetool repair. First thing I did was to drop that nice, but ultimately unnecessary, index. That stopped to repair crashes, but repair would take days to complete. Second, I boosted the 8GB to 24GB. Sounds like this is not a luxury you have, but that is what it took. That changed the repair time from days to hours, like 8 of them. Third, I upgraded from 2.0 to 2.2. Once I got repair run on all three nodes, took 24hrs, I upgraded each node, one at a time, and then ran repair again on each node after all were upgraded. Repair now, not only does not crash, but it completes across the whole cluster in about two hours. So much faster, much more stable. I have since added a fourth node and a second replica. Still no problems. I think the biggest issue was that secondary index. I also found that installing jemalloc was a huge speed boost.

Upvotes: 0

Aristotle Allen
Aristotle Allen

Reputation: 63

Inserts should simply be getting flushed out to disk, not causing OOM exceptions.

Cassandra does require lots of memory though, 2GB seems very low. Its performance comes from not only having a lot of memory per node, but lots of nodes, creative a very large cache.

I would recommend you have an 8GB heap per node, and your VMS should be bumped up to ~32GB of memory. Make sure you have JNA installed so Cassandra can leverage the additional off heap memory.

Upvotes: 1

Related Questions