CODEWITHSUNDEEP
cassandra
Asad
Asad

Reputation: 35

Cassandra: too many sstables - node goes down instantly

I'm in situation where too many SSTables (98,000+) are created for a single table and many more for other CFs. Node keeps crashing complaining insufficient memory for jre. I've tried increasing linux nofile limit to 200K and max_heap_size to 16G but no avail!

Looking for help to know ways as how i can reduce # of SSTables (compaction?) and keep node up so to do the maintenance.

Thanks in advance!

errors:

There is insufficient memory for the Java Runtime Environment to continue. Out of Memory Error (os_linux.cpp:2627), pid=22667, tid=139622017013504

--------------- T H R E A D ---------------

Current thread (0x00007efc78b83000): JavaThread "MemtableFlushWriter:2" daemon [_thread_in_vm, id=22726, stack(0x00007efc48b61000,0x00007efc48ba2000)]

Stack: [0x00007efc48b61000,0x00007efc48ba2000], sp=0x00007efc48b9f730, free space=249k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xab97ea] VMError::report_and_die()+0x2ba V [libjvm.so+0x4f9dcb] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b V [libjvm.so+0x91a7c3] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103 V [libjvm.so+0x91ad19] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29 V [libjvm.so+0x91502a] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a

JRE version: Java(TM) SE Runtime Environment (8.0_65-b17) (build 1.8.0_65-b17)

Upvotes: 1

Views: 672

Answers (1)

Marko Švaljek
Marko Švaljek

Reputation: 2101

I would treat this as a dead node situation:

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html

https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsReplaceNode.html

After you finish the procedure the node will have way less sstables etc. The thing that is bothering me is how did you come into this situation. Can you provide some schema, insert, delete, ttl related info and describe the workload?

Upvotes: 1

Related Questions