CODEWITHSUNDEEP
cassandracqlshnodetool
user_1357
user_1357

Reputation: 7940

Apache Cassandra: Nodetool Stats Which shows Memtable Flush Frequency

Currently I am debugging performance issue with Apache Cassandra. when Memtable for the column family is filled, it is queued to be flushed to SSTable. This flushing happens often when you perform massive writes. When this queue is filled up, writes are blocked until next successful completion of flush. This indicates that your node cannot handle writes it is receiving.

Is there a matrix in nodetool indicating this behaviour? In other words, I want a data indicating a node cannot keep up with writes it is receiving.

Thanks!!

Upvotes: 2

Views: 1178

Answers (3)

doanduyhai
doanduyhai

Reputation: 8812

I want a data indicating a node cannot keep up with writes it is receiving.

Your issue is likely bound to disk I/O not being able to handle the throughput --> flushes of memtables queue up --> writes are blocked

the command dstat is your friend to investigate I/O issues. Some others linux commands may be also handy. Read this excellent blog post from Amy Tobey: https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html

Is there a matrix in nodetool indicating this behaviour?

nodetool tpstats

Upvotes: 3

Chris Lohfink
Chris Lohfink

Reputation: 16400

Thats not really true for a couple years. The active memtable is switched and a new memtable takes its position as live. New mutations occur on this live memtable while the "to be flushed" memtables are included in local reads. The MemtableFlushWriter thread pool has the flush tasks queued on it. So you can see how many are pending there (under tpstats). The mutations backing up you can also see under the MutationStage.

Ultimately

nodetool tpstats

Is likely what your looking for.

Upvotes: 4

MarcintheCloud
MarcintheCloud

Reputation: 1653

I believe you're looking for tp (thread pool) stats. 

nodetool tpstats

Typically blocked FlushWriters indicates that your storage system is having trouble keeping up with the write workload. Are you using spinning disks by chance? You'll also want to keep an eye on iostat in this case as well.

Here's the docs for tpstats: https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsTPstats.html

Upvotes: 2

Related Questions