datastax-startup FSReadError HintsDescriptor.readFromFile

Question

DataStax Enterprise Server 5.1.2

In a 5-node cluster one node crashed. DSE will not start on this node. Here is the error from system log:

ERROR [main] 2018-07-13 12:47:37,336 CassandraDaemon.java:705 - Exception encountered during startup org.apache.cassandra.io.FSReadError: java.io.EOFException at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:235) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_141] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_141] at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_141] at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_141] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_141] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_141] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_141] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_141] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_141] at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at org.apache.cassandra.hints.HintsService.(HintsService.java:97) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at org.apache.cassandra.hints.HintsService.(HintsService.java:88) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at org.apache.cassandra.hints.HintsService.(HintsService.java:66) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at org.apache.cassandra.service.StorageProxy.(StorageProxy.java:129) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at java.lang.Class.forName0(Native Method) ~[na:1.8.0_141] at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_141] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:638) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:623) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:393) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:465) ~[dse-core-5.1.2.jar:5.1.2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:599) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at com.datastax.bdp.DseModule.main(DseModule.java:93) [dse-core-5.1.2.jar:5.1.2] Caused by: java.io.EOFException: null at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803) ~[na:1.8.0_141] at org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:344) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758] at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:231) ~[cassandra-all-3.11.0.1758.jar:3.11.0.1758]

Can someone from datastax please advise? I have been unable to find any useful suggestions on the web. Thanks for your time.

RL

Answer 1

I located the offending hints' files by referring to datastax default directories page. In my case they were in /var/lib/cassandra/hints. There were two 0-byte files in that directory. I created a temp directory, moved them to that, and started dse with no problem.

For some unknown (to me) reason I had to reinstall all opscenter agents used in this 5-node cluster (opscenter 6.1.5) and repair took around 20 hours since this node had been offline for around 8 days.