CODEWITHSUNDEEP
cassandra
gerrymcdev
gerrymcdev

Reputation: 111

cassandra Exception encountered during startup: index (1) must be less than size (1)

I experienced a power outage which killed all 4 the nodes in my Cassandra cluster. I've brought back all the boxes, but now when I try to start Cassandra (method bin/Cassandra) I am getting the following message on 3 out of 4 after there is a bit of replaying the log.

Exception encountered during startup: index (1) must be less than size (1)
 INFO 10:11:00,479 CFS(Keyspace='system', ColumnFamily='peers') liveRatio is 13.10204081632653 (just-counted was 13.10204081632653).  calculation took 18ms for 21 columns

(that's just the end, I can post more message if it helps)

The one that came up is running for reads only, I assume as I have replication factor 3 on all keyspaces so it can't work without the others.

Any idea's as to what to do to retrieve the situation.. I have quite a bit of valuable information in the database.

Cassandra v1.2.6,

CentOs v6.4

Update:

All three dead nodes seem to be failing during replay of the commit log.. I've bolded the line they get to. My initial thoughts are to remove that commit log and try restarting.. but I'm a little afraid of the consequences!!

 INFO 10:46:48,600 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208294.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208295.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208296.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208297.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208298.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208299.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208301.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208302.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208304.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208305.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208306.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208307.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208308.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208309.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208310.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208311.log, /var/lib/cassandra/commitlog/CommitLog-2-1381251642075.log, /var/lib/cassandra/commitlog/CommitLog-2-1381251731119.log
 INFO 10:46:48,614 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log
 INFO 10:46:49,598 GC for ParNew: 605 ms for 2 collections, 50055912 used; max is 1046937600
 INFO 10:46:51,775 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log
 INFO 10:46:51,776 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log
 INFO 10:46:53,995 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log
 INFO 10:46:53,995 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log
 **INFO 10:46:54,087 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log**
ERROR 10:46:54,088 Exception encountered during startup
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
    at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
    at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
    at java.util.TreeMap.compare(TreeMap.java:1188)
    at java.util.TreeMap.put(TreeMap.java:531)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
    at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:397)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:202)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:97)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:298)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484)
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
    at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
    at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
    at java.util.TreeMap.compare(TreeMap.java:1188)
    at java.util.TreeMap.put(TreeMap.java:531)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
    at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:397)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:202)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:97)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:298)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484)
Exception encountered during startup: index (1) must be less than size (1)
 INFO 10:46:54,110 CFS(Keyspace='system', ColumnFamily='peers') liveRatio is 13.10204081632653 (just-counted was 13.10204081632653).  calculation took 19ms for 21 columns

Upvotes: 2

Views: 3265

Answers (2)

Praful Bagai
Praful Bagai

Reputation: 17402

Although its very late and you must have solve this issue as well. For others having similar problem, try this

sudo bash -c 'rm -rf /var/lib/cassandra/data/system/*'

Remember, its only for development purposes. THIS COMMAND WILL DELETE ALL YOUR DATA.

Upvotes: 0

jbellis
jbellis

Reputation: 19377

My best theory is that this is a possible consequence of dropping and re-creating a table with the same name (https://issues.apache.org/jira/browse/CASSANDRA-5905). We are targeting a fix for 2.1 (https://issues.apache.org/jira/browse/CASSANDRA-5202); in the meantime, prefer TRUNCATE over drop/recreate.

Upvotes: 2

Related Questions