CODEWITHSUNDEEP
timeoutcassandracassandra-2.0
user1200518
user1200518

Reputation: 1

Cassandra rpc timeout occurring during heavy writes and deletes

I'm using cassandra 2.0, and I have created a column family that looks like this:

CREATE TABLE user_id_timestamp_index (
  user_id int,
  timestamp text,
  PRIMARY KEY (user_id, timestamp)
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'LeveledCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

I've written over 2 million rows to this table without any issues, and I perform numerous deletes as well.

The problem arises after about 10k deletes or so in rapid succession, and I start encountering numerous rpc_timeouts. A simple "delete from user_id_timestamp_index where user_id = 5 AND timestamp = '12345'" via cqlsh fails during this period.

Things I've noticed and tried:

  1. During rpc timeouts, Load Average on 2 nodes (out of 5) shoot up to about 50.
  2. Compaction is done almost every 5 minutes during these load-intensive writes and deletes.
  3. During rpc_timeouts, tpstats shows pending mutation stages: MutationStage 64 (active) 395 (pending) 48182373 (completed) 0 0
  4. Timeouts tend to occur when memtable data size exceeds 3 mb for this CF.
  5. After I perform a nodetool flush, pending mutations goes to zero and rpc time disappears, until memtable size creeps up again to past 3 mb.

My question is, there a configuration I can tune? For example, is the solution to simply force a memtable flush on this column family every 5 minutes? Lessen the write load on this table? A way to have faster writes and reduce pending stages? Or is there a better solution?

Upvotes: 0

Views: 544

Answers (1)

jbellis
jbellis

Reputation: 19377

If you're experiencing GC pressure (which you can tell from the GCInspector lines in the log), you can reduce the amount of memory used by memtables by adjusting memtable_total_space_in_mb in cassandra.yaml. You may also need to reduce the key or row cache settings instead or as well.

Upvotes: 0

Related Questions