cassandra sstableloader hanging with receiving progress 100%

Question

I'm trying to load data by sstableloader in cassandra 2.0.7. The terminal shows progress 100%. I check netstats by nodetool netstats It shows:

Mode: NORMAL
Bulk Load 21d7d610-a5f2-11e5-baa7-8fc95be03ac4
    /10.19.150.70
        Receiving 4 files, 138895248 bytes total
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-8-Data.db 67039680/67039680 bytes(100%) received from /10.19.150.70
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-10-Data.db 3074549/3074549 bytes(100%) received from /10.19.150.70
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-9-Data.db 43581052/43581052 bytes(100%) received from /10.19.150.70
            /root/data/whatyyun/metadata/whatyyun-metadata-tmp-jb-7-Data.db 25199967/25199967 bytes(100%) received from /10.19.150.70
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Commands                        n/a         0              0
Responses                       n/a         0          11671

The sstableloader hangs for hours. I check the log there is an error that may concerns.

ERROR [CompactionExecutor:7] 2015-12-19 09:45:53,811 CassandraDaemon.java (line 198) Exception in thread Thread[CompactionExecutor:7,1,main]
java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compareTimestampBytes(TimeUUIDType.java:62)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:51)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:31)
        at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:36)
        at java.util.concurrent.ConcurrentSkipListMap.findNode(ConcurrentSkipListMap.java:804)
        at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:828)
        at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1626)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215)
        at org.apache.cassandra.db.Memtable.put(Memtable.java:173)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:893)
        at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex.insert(AbstractSimplePerColumnSecondaryIndex.java:107)
        at org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(SecondaryIndexManager.java:441)
        at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:407)
        at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:833)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [NonPeriodicTasks:1] 2015-12-19 09:45:53,812 CassandraDaemon.java (line 198) Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
        at org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:142)
        at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:113)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
        ... 9 more
Caused by: java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Buffer.java:532)
        at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compareTimestampBytes(TimeUUIDType.java:62)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:51)
        at org.apache.cassandra.db.marshal.TimeUUIDType.compare(TimeUUIDType.java:31)
        at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85)
        at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:36)
        at java.util.concurrent.ConcurrentSkipListMap.findNode(ConcurrentSkipListMap.java:804)
        at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:828)
        at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1626)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215)
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1626)
        at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215)
        at org.apache.cassandra.db.Memtable.put(Memtable.java:173)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:893)
        at org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex.insert(AbstractSimplePerColumnSecondaryIndex.java:107)
        at org.apache.cassandra.db.index.SecondaryIndexManager.indexRow(SecondaryIndexManager.java:441)
        at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:407)
        at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
        at org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:833)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        ... 3 more

The scheme of the table is as follows:

CREATE TABLE metadata (
  userid timeuuid,
  dirname text,
  basename text,
  ctime timestamp,
  fileid timeuuid,
  imagefileid timeuuid,
  imagefilesize int,
  mtime timestamp,
  nodetype int,
  showname text,
  size bigint,
  timelong text,
  PRIMARY KEY (userid, dirname, basename, ctime)
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

CREATE INDEX idx_fileid ON metadata (fileid);

CREATE INDEX idx_nodetype ON metadata (nodetype);

Can I kill the process of the sstableloader safely? Has this bulk load process finished?

Answer 1

From the above error it seems you are running some resource crunch. So you need to tune some setting on memory side like Xmx,Xms Which indicates min and max heap size in cassandra-env.sh file for lower version of Cassandra(i.e 2.x)

After tuning above you need to restart you node/cluster and try loading again.

Answer 2

You should try to increase heap for SSTABLELoader.

vim $(which sstableloader)

#########"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -Xmx$MAX_HEAP_SIZE \
"$JAVA" $JAVA_AGENT -ea -cp "$CLASSPATH" $JVM_OPTS -XX:+UseG1GC -Xmx10G -Xms10G -XX:+UseTLAB -XX:+ResizeTLAB \
        -Dcassandra.storagedir="$cassandra_storagedir" \
        -Dlogback.configurationFile=logback-tools.xml \
        org.apache.cassandra.tools.BulkLoader "$@"

I hope that would solve your issue.

Answer 3

Your node must be running out of resources may be due to heavy load or any other process. Try restarting Cassandra on the nodes running on high load and see if that helps.