CODEWITHSUNDEEP
javacassandra
Neuron
Neuron

Reputation: 71

Cassandra shutdown caused by: java.io.SyncFailedException: sync failed

I am using Cassnadra 2.1.13 single node cluster and database was running all good without any issues.

Then there was this following exception because of which Cassandra service was brought down.

In a single node cluster what could be the reason for these exceptions specially sync failed, any hints or pointers to get to the root cause of this issue?

INFO [MemtableFlushWriter:22] Memtable.java:347 - Writing Memtable-compaction_history@1884003282(0.521KiB serialized bytes, 19 ops, 0%/0% of on/off-heap limit) INFO [MemtableFlushWriter:22] Memtable.java:382 - Completed flushing /myspace/db/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-tmp-ka-8-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1540762960100, position=2642365) ERROR [MemtableFlushWriter:21] CassandraDaemon.java:229 - Exception in thread Thread[MemtableFlushWriter:21,5,main] org.apache.cassandra.io.FSWriteError: java.io.SyncFailedException: sync failed at org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:254) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:263) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.io.util.SequentialWriter.close(SequentialWriter.java:451) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.io.compress.CompressedSequentialWriter.close(CompressedSequentialWriter.java:264) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:503) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:455) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:447) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:442) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:389) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:335) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.13.jar:2.1.13] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1120) ~[apache-cassandra-2.1.13.jar:2.1.13] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_192] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_192] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_192] Caused by: java.io.SyncFailedException: sync failed at java.io.FileDescriptor.sync(Native Method) ~[na:1.8.0_192] at org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:250) ~[apache-cassandra-2.1.13.jar:2.1.13] ... 15 common frames omitted

ERROR [MemtableFlushWriter:21] StorageService.java:453 - Stopping gossiper WARN [MemtableFlushWriter:21] StorageService.java:359 - Stopping gossip by operator request INFO [MemtableFlushWriter:21] Gossiper.java:1455 - Announcing shutdown

After restarting the Cassandra service there are no issues and working as expected.

Upvotes: 1

Views: 1098

Answers (1)

Alex Ott
Alex Ott

Reputation: 87339

I suspect that this is either:

  • disk issue - check the /var/log/messages and SMART status of the disk
  • disk was full
  • file was removed (although that's very improbable, but could happen if you have antivirus on the system)

Upvotes: 1

Related Questions