CODEWITHSUNDEEP
exceptioncassandracluster-computingeofcassandra-0.7
samarth
samarth

Reputation: 4024

While Cassandra compaction Fatal exception in thread CompactionExecutor

  1. I am having cassandra cluster of 12 nodes on EC2 running cassandra-0.8.2. While compaction I got the following exception which caused Seed node to get down.

  2. Below is the exception stack trace.

ERROR [CompactionExecutor:31] 2011-12-16 08:06:02,308 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:31,1,main] java.io.IOError: java.io.EOFException: EOF after 430959023 bytes out of 778986868 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:90) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:74) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39) at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284) at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326) at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:69) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:569) at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:141) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:107) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException: EOF after 430959023 bytes out of 778986868 at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:229) at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:63) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.(SSTableIdentityIterator.java:141) ... 23 more

Upvotes: 1

Views: 1614

Answers (1)

jbellis
jbellis

Reputation: 19377

It looks like this is indeed caused by corrupt sstables (which may indicate a hardware problem). My recommendations:

  1. Upgrade to the latest stable 0.8.x version of Cassandra. This will be a drop-in replacement for 0.8.2.
  2. Run "nodetool scrub" on the machine having problems
  3. Review http://www.datastax.com/docs/1.0/install/cluster_init -- I recommend two seed nodes per data center, but remember that a seed node is only consulted when restarting nodes, so it's not a big deal to have one down during normal operation

Upvotes: 2

Related Questions