CODEWITHSUNDEEP
cassandraout-of-memory
alexadr
alexadr

Reputation: 113

Why cassandra fails with OutOfMemoryError during sstables compaction

Hi may be this is a stupid question, but I did not find the answer via google.

So what I have:

I've created next test column family:

CREATE COLUMN FAMILY TEST_HUGE_SF
    WITH comparator = UTF8Type
    AND key_validation_class=UTF8Type;

Then I try to insert rows in this column family. I use astyanax lib to access cassandra:

    final long START = 1;
    final long MAX_ROWS_COUNT = 1000000000; // 1 Billion

    Keyspace keyspace = AstyanaxProvider.getAstyanaxContext().getClient();

    ColumnFamily<String, String> cf = new ColumnFamily<>(
        "TEST_HUGE_SF", 
        StringSerializer.get(), 
        StringSerializer.get());

    MutationBatch mb = keyspace.prepareMutationBatch()
            .withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
    for (long i = START; i<MAX_ROWS_COUNT; i++) {
        long t = i % 1000;
        if (t == 0) {
            System.out.println("pushed: " + i);
            mb.execute();
            Thread.sleep(1);
            mb = keyspace.prepareMutationBatch()
                    .withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
        }

        ColumnListMutation<String> clm = mb.withRow(cf, String.format("row_%012d", i));
        clm.putColumn("col1", i);
        clm.putColumn("col2", t);
    }
    mb.execute();

So as you can see from code, I try to insert 1 Billion rows, each one contains two columns, each column contains simple long value.

After inserting ~ 122 million rows, - cassandra crashed with OutOfMemoryError. In logs there is next:

 INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,334 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941,].  865 252 169 bytes to 901 723 715 (~104% of original) in 922 963ms = 0,931728MB/s.  26 753 257 total rows, 26 753 257 unique.  Row merge counts were {1:26753257, 2:0, 3:0, 4:0, }
 INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,337 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1571] 2014-08-08 08:31:46,167 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1571,1,main]
java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
    at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
    at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
    at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
    at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
    at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
    at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
    at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
 INFO [CompactionExecutor:1570] 2014-08-08 08:31:46,994 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-3213,].  34 773 524 bytes to 35 375 883 (~101% of original) in 44 162ms = 0,763939MB/s.  1 151 482 total rows, 1 151 482 unique.  Row merge counts were {1:1151482, 2:0, 3:0, 4:0, }
 INFO [CompactionExecutor:1570] 2014-08-08 08:31:47,105 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1570] 2014-08-08 08:31:47,110 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1570,1,main]
java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
    at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
    at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
    at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
    at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
    at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
    at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
    at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

As I see cassandra crashes during sstables compaction.

Does this mean that to handle more rows cassandra needs more heap space?

I expected that lack of heap space will only affect performance. Can someone describe, why my expectations are wrong?

Upvotes: 1

Views: 5664

Answers (1)

Max Kotasek
Max Kotasek

Reputation: 21

Someone else noted this - 1GB heap is very small. With Cassandra 2.0, you could look into this tuning guide for further information: http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_tune_jvm_c.html

Another consideration is how garbage collecting is being handled. In the cassandra log directory, there should be also GC logs indicating how often and how long the collections were. You can monitor them live using jvisualvm, if you want.

Upvotes: 1

Related Questions