How can I fix a Cassandra RuntimeException: [hash] is not defined as a collection?

Question

I've been experiencing problems with a 3 node, RF 3, Cassandra 2.1.16 cluster and will try and describe them as accurately as possible considering the knowledge I've got at this point.

Adding a new row to a table either:

a) does not replicate to node 3 until I do a nodetool repair or

b) it replicates the row to node 3 but leaves some columns with null or outdated values.

Might be unrelated but this seems to happen in rows where you first insert, and then do reads and writes, not in the append-only tables.

I'm aware that Cassandra doesn't like to have column names or schema names reused as explained in another question, and I adapted my delivery pipeline to just create newly named structures, also that there seems to be a loosely coupled relationship between data and schemas - sometimes schemas don't match the data.

I added a few new tables and columns to existing tables and now I've got this coming up each couple a minutes on node 3 only, clearly related with compaction:

INFO  [CompactionExecutor:48525] 2017-03-27 11:08:04,851 CompactionTask.java:141 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7719-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7722-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7721-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7720-Data.db')]
INFO  [CompactionExecutor:48524] 2017-03-27 11:08:04,851 CompactionTask.java:141 - Compacting [SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-3-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-1-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-2-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-5-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-4-Data.db'), SSTableReader(path='/var/lib/cassandra/data/KEYSPACE/TABLENAME-f8195440e63a11e6bfad753f1c36fea2/KEYSPACE-TABLENAME-ka-462-Data.db')]
INFO  [CompactionExecutor:48525] 2017-03-27 11:08:04,866 CompactionTask.java:274 - Compacted 4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-7723,].  476 bytes to 185 (~38% of original) in 14ms = 0.012602MB/s.  3 total partitions merged to 1.  Partition merge counts were {1:1, 2:1, }
INFO  [CompactionExecutor:48524] 2017-03-27 11:08:04,934 ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 148 (0%) on-heap, 20 (0%) off-heap
INFO  [MemtableFlushWriter:6333] 2017-03-27 11:08:04,934 Memtable.java:347 - Writing Memtable-compactions_in_progress@1966348368(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
INFO  [MemtableFlushWriter:6333] 2017-03-27 11:08:04,935 Memtable.java:382 - Completed flushing /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-7724-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1485986982740, position=11004802)
ERROR [CompactionExecutor:48524] 2017-03-27 11:08:04,954 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExecutor:48524,1,main]
java.lang.RuntimeException: 6865617274707265736372697074696f6e73 is not defined as a collection
    at org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.composites.CompoundSparseCellNameType$WithCollection.compare(CompoundSparseCellNameType.java:287) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.composites.AbstractCellNameType$4.compare(AbstractCellNameType.java:82) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.composites.AbstractCellNameType$4.compare(AbstractCellNameType.java:79) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.utils.MergeIterator$Candidate.compareTo(MergeIterator.java:154) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.utils.MergeIterator$Candidate.compareTo(MergeIterator.java:131) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at java.util.PriorityQueue.siftUpComparable(PriorityQueue.java:656) ~[na:1.8.0_121]
    at java.util.PriorityQueue.siftUp(PriorityQueue.java:647) ~[na:1.8.0_121]
    at java.util.PriorityQueue.offer(PriorityQueue.java:344) ~[na:1.8.0_121]
    at java.util.PriorityQueue.add(PriorityQueue.java:321) ~[na:1.8.0_121]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
    at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
    at org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:174) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:193) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-2.1.16.jar:2.1.16]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

I dumped a "describe keyspace KEYSPACENAME;" from each node to a file and did a diff, all 3 files are identical. Also did a nodetool repair on each node.

On a side note, after I made the changes to the tables, I found some massive clock drifts (4-15 seconds) and then fixed the ntp service configurations also by placing the nodes as ntp peers.

How can I fix a Cassandra RuntimeException: [hash] is not defined as a collection?

Answers (1)

Related Questions