CODEWITHSUNDEEP
performancecassandralatency
Tipok
Tipok

Reputation: 695

Cassandra READ operations latency dependency of WRITE operations

I'm testing Cassandra performance in the case of simultaneous read and writes operations.

Latency for READ operations is important for me, WRITES usually are done once per day, so I'm investigating if the WRITE operations would affect READ operations latency.

I have a table of 10_000 records which I READ from application at 50 RPS. For example, the latency is 10ms, REQUEST_SERIAL_CONSISTENCY and REQUEST_CONSISTENCY is QUORUM for READ operations.

Query for READs is: SELECT * from table where id = X

Simultaneously I run another application which insert same records to table (with same ids, so in fact it is UPDATE operations), the REQUEST_SERIAL_CONSISTENCY and REQUEST_CONSISTENCY is ALL for WRITE operations. In several threads, I load my Cassandra cluster with batches of INSERT queries size of 60 queries in batch.

Query for WRITEs is: INSERT into table values(id,..) where id = X;

What I expect it is the latency increasing for READ operations when WRITES is enabled. But latency is stable in general besides random spikes like:

enter image description here

Why hasn't latency been changed, was it wrong to expect uniform latency increase? What about these spikes, can it be connected with MemTable flushes to disk? I can see them in statistic:

nodetool tpstats
MemtableFlushWriter          0      0       4272      0       0

Info of my cluster:

cqlsh> SELECT * FROM system_schema.keyspaces;

 keyspace_name      | durable_writes | replication
--------------------+----------------+-------------------------------------------------------------------------------------
        system_auth |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}
      system_schema |           True |                             {'class': 'org.apache.cassandra.locator.LocalStrategy'}
 system_distributed |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
             system |           True |                             {'class': 'org.apache.cassandra.locator.LocalStrategy'}
             search |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}
      system_traces |           True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '2'}

Upvotes: 0

Views: 890

Answers (1)

Manish Khandelwal
Manish Khandelwal

Reputation: 2310

With 10000 records, I do not expect much change in latency. Latency can get hit due to lot of factor. It could be that Cassandra is doing I/O for compactions, repairs and that could hit your latency even when no write is done. So I assume random spikes you are seeing is because of some compactions running in background. One more thing, when you say you are inserting records in batch of 60, please dont use Cassandra batch for inserting records as they will be slow.

Upvotes: 1

Related Questions