Cassandra exits with no errors

Question

Linux x64, Cassandra 2, Sun Java 1.7. After some reads - a bit more intensive than usual - cassandra simply dies with nothing helpful in the logfile.

This is how it is started (I used ps aux | grep cassandra):

109      29705 14.1 24.5 102272904 12122668 ?  SLl  10:38   0:42 java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=1000003 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar:/usr/share/cassandra/apache-cassandra-2.0.6.jar:/usr/share/cassandra/apache-cassandra-thrift-2.0.6.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar: -XX:HeapDumpPath=/var/lib/cassandra/java_1400575116.hprof -XX:ErrorFile=/var/lib/cassandra/hs_err_1400575116.log org.apache.cassandra.service.CassandraDaemon

This is the log (full!):

INFO [ScheduledTasks:1] 2014-05-19 17:01:09,489 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-compaction_history@1762312178(253/2530 serialized/live bytes, 9 ops)
 INFO [FlushWriter:2] 2014-05-19 17:01:09,489 Memtable.java (line 331) Writing Memtable-compaction_history@1762312178(253/2530 serialized/live bytes, 9 ops)
 INFO [ScheduledTasks:1] 2014-05-19 17:01:09,584 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-schema_keyspaces@141563727(113/1130 serialized/live bytes, 4 ops)
 INFO [ScheduledTasks:1] 2014-05-19 17:01:09,613 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-schema_columnfamilies@1997162460(49256/492560 serialized/live bytes, 1151 ops)
 INFO [ScheduledTasks:1] 2014-05-19 17:01:09,647 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-schema_columns@1140062610(66556/665560 serialized/live bytes, 1863 ops)
 INFO [FlushWriter:2] 2014-05-19 17:01:09,843 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/compaction_history/system-compaction_history-jb-415-Data.db (238 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
 INFO [FlushWriter:2] 2014-05-19 17:01:09,845 Memtable.java (line 331) Writing Memtable-schema_keyspaces@141563727(113/1130 serialized/live bytes, 4 ops)
 INFO [FlushWriter:2] 2014-05-19 17:01:10,083 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-108-Data.db (135 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
 INFO [FlushWriter:2] 2014-05-19 17:01:10,084 Memtable.java (line 331) Writing Memtable-schema_columnfamilies@1997162460(49256/492560 serialized/live bytes, 1151 ops)
 INFO [CompactionExecutor:23] 2014-05-19 17:01:10,084 CompactionTask.java (line 115) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-106-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-107-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-105-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-108-Data.db')]
 INFO [CompactionExecutor:23] 2014-05-19 17:01:10,618 CompactionTask.java (line 275) Compacted 4 sstables to [/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-109,].  793 bytes to 334 (~42% of original) in 531ms = 0,000600MB/s.  7 total partitions merged to 4.  Partition merge counts were {1:2, 2:1, 3:1, }
 INFO [FlushWriter:2] 2014-05-19 17:01:10,631 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-107-Data.db (10262 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
 INFO [FlushWriter:2] 2014-05-19 17:01:10,632 Memtable.java (line 331) Writing Memtable-schema_columns@1140062610(66556/665560 serialized/live bytes, 1863 ops)
 INFO [FlushWriter:2] 2014-05-19 17:01:10,926 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-96-Data.db (14011 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=226683)
 INFO [CompactionExecutor:17] 2014-05-19 17:01:10,927 CompactionTask.java (line 115) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-93-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-95-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-94-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-96-Data.db')]
 INFO [CompactionExecutor:17] 2014-05-19 17:01:11,267 CompactionTask.java (line 275) Compacted 4 sstables to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-97,].  23 264 bytes to 14 011 (~60% of original) in 337ms = 0,039650MB/s.  10 total partitions merged to 4.  Partition merge counts were {2:3, 4:1, }
 INFO [ScheduledTasks:1] 2014-05-19 18:01:09,489 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-compaction_history@683113558(661/6610 serialized/live bytes, 21 ops)
 INFO [FlushWriter:3] 2014-05-19 18:01:09,490 Memtable.java (line 331) Writing Memtable-compaction_history@683113558(661/6610 serialized/live bytes, 21 ops)
 INFO [FlushWriter:3] 2014-05-19 18:01:09,704 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data/system/compaction_history/system-compaction_history-jb-416-Data.db (372 bytes) for commitlog position ReplayPosition(segmentId=1400508070263, position=227709)
 INFO [CompactionExecutor:31] 2014-05-19 20:01:10,471 AutoSavingCache.java (line 250) Saved KeyCache (63 items) in 1192 ms
 INFO [CompactionExecutor:32] 2014-05-20 00:01:10,040 AutoSavingCache.java (line 250) Saved KeyCache (63 items) in 761 ms
 INFO [CompactionExecutor:33] 2014-05-20 04:01:10,280 AutoSavingCache.java (line 250) Saved KeyCache (163 items) in 1000 ms
 INFO [CompactionExecutor:34] 2014-05-20 08:01:10,704 AutoSavingCache.java (line 250) Saved KeyCache (220 items) in 1424 ms
 INFO [StorageServiceShutdownHook] 2014-05-20 10:36:12,376 ThriftServer.java (line 141) Stop listening to thrift clients
 INFO [StorageServiceShutdownHook] 2014-05-20 10:36:12,389 Server.java (line 181) Stop listening for CQL clients
 INFO [StorageServiceShutdownHook] 2014-05-20 10:36:12,390 Gossiper.java (line 1251) Announcing shutdown
 INFO [StorageServiceShutdownHook] 2014-05-20 10:36:14,391 MessagingService.java (line 667) Waiting for messaging service to quiesce
 INFO [ACCEPT-localhost/127.0.0.1] 2014-05-20 10:36:14,391 MessagingService.java (line 907) MessagingService has terminated the accept() thread

As you see, it shows "Gossiper.java (line 1251) Announcing shutdown" - but why?! This is really annoying and since it happens in production, this is business-critical =(

Answer 1

I added auto_bootstrap:false in my cassandra.yaml file and it works for me after restarting Cassandra.

Answer 2

this could happen because of an out of memory error. trying running dmesg and see actually why it is killed.

Answer 3

I've had this happen to me, but this with an image deployed onto a VMWare Player. It turned out that it was because the SSD drive on which the Player was hosted on ran out of disk space. There were no related / helpful warnings in the log files in my case, either. Everything was fine after I cleared up some space.