CODEWITHSUNDEEP
cassandra
Caggy
Caggy

Reputation: 41

Cassandra exceptions during repair

I am currently having issues with cassandra when running a nodetool repair.

I ran a nodetool repair on each of our cassandra nodes and came across the exception mentioned below.

After doing some reading I ran a "nodetool scrub" and "sstablescrub" with no success, I am still seeing the same errors when running a nodetool repair.

We are using cassandra version 2.0.1.

Has anyone else seen this problem?

ERROR [AntiEntropySessions:3] 2013-11-26 15:25:03,315 RepairSession.java (line 278) [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c] session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]] Validation failed in /<ipaddress-1>
        at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
        at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
ERROR [AntiEntropySessions:3] 2013-11-26 15:25:03,315 CassandraDaemon.java (line 185) Exception in thread Thread[AntiEntropySessions:3,5,RMI Runtime]
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException: [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]] Validation failed in /<ipaddress-1>
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]] Validation failed in /<ipaddress-1>
        at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
        at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
        ... 3 more
 INFO [AntiEntropySessions:4] 2013-11-26 15:25:03,319 RepairSession.java (line 236) [repair #fa075b20-56f1-11e3-9885-5938b4e97c9c] new session: will sync /<ipaddress1>, /<ipaddress2>, /<ipaddress3> on range (2637939872511762724,2642446772090452246] for <keyspace>.[<table-1>, <table-2>, <table-3>, <table-4>, <table-5>]
 INFO [AntiEntropySessions:4] 2013-11-26 15:25:03,319 RepairJob.java (line 116) [repair #fa075b20-56f1-11e3-9885-5938b4e97c9c] requesting merkle trees for <table-2> (to [/<ipaddress1>, <ipaddress2>, <ipaddress3>])
ERROR [ValidationExecutor:5] 2013-11-26 15:25:03,353 Validator.java (line 242) Failed creating a merkle tree for [repair #f9f86700-56f1-11e3-9885-5938b4e97c9c on <keyspace>/<table-1>, (-3566327001497837731,-3559225618918749690]], /<ipaddress-2> (see log for details)
ERROR [ValidationExecutor:5] 2013-11-26 15:25:03,353 CassandraDaemon.java (line 185) Exception in thread Thread[ValidationExecutor:5,1,main]
java.lang.AssertionError
        at org.apache.cassandra.db.compaction.PrecompactedRow.update(PrecompactedRow.java:171)
        at org.apache.cassandra.repair.Validator.rowHash(Validator.java:198)
        at org.apache.cassandra.repair.Validator.add(Validator.java:151)
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:798)
        at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:60)
        at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:395)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Upvotes: 4

Views: 2388

Answers (1)

zznate
zznate

Reputation: 1908

Depending on your data model, you may be seeing the effects CASSANDRA-6152. See this comment for details:

https://issues.apache.org/jira/browse/CASSANDRA-6152?focusedCommentId=13793977&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13793977

Regardless, you should update to 2.0.3 immediately as there are several other fairly severe bugs lurking on your current version.

Upvotes: 3

Related Questions