CODEWITHSUNDEEP
cassandra
Atmesh Mishra
Atmesh Mishra

Reputation: 549

Cassandra read performance degrade as we increase data on nodes

Problem: I loaded our servers with 50,000 users and each user had 1000 records initially and after sometime, 20 more records were added to each users. I wanted to fetch the 20 additional records that were added later(Query : select * from table where userID='xyz' and timestamp > 123) here user_id and timestamp are part of primary key. It worked fine when I had only 50,000 users. But as soon as I added another 20GB of dummy data, the performance for same query i.e. fetch 20 additional records for 50,000 users dropped significantly. Read performance is getting degraded with increase in data. As far as I have read, this should not have happened as keys get cached and additional data should not matter.

what could be possible cause for this? CPU and RAM utilisation is negligible and I cant find out what is causing the query time to increase. I have tried changing compaction strategy to "LeveledCompaction" but that didn't work either.

EDIT 1

output for dstat command

EDIT 2 Heap size is 8GB. The 20GB data is added in a way similar to the way in which the initial 4GB data was added (the 50k userIDs) and this was done to simulate real world scenario. "userID" and "timestamp" for the 20GB data is different and is generated randomly. Scenario is that I have 50k userIDs with 1020 rows where 1000 rows were added first and then additional 20 rows were added after some timestamp, I am fetching these 20 messages. It works fine if only 50k userIDs are present but once I have more userIDs (additional 20GB) and I try to fetch those same 20 messages (for initial 50k userIDs), the performance degrades.
JMX output

EDIT 3 cassandra.yaml

Upvotes: 3

Views: 1457

Answers (1)

xmas79
xmas79

Reputation: 5180

Read performance is getting degraded with increase in data.

This should only happen when your add a lot of records in the same partition.

From what I can understand your table may looks like:

CREATE TABLE tbl (
    userID text,
    timestamp timestamp,
    ....
    PRIMARY KEY (userID, timestamp)
);

This model is good enough when the volume of the data in a single partition is "bound" (eg you have at most 10k rows in a single partition). The reason is that the coordinator gets a lot of pressure when dealing with "unbound" queries (that's why very large partitions are a big no-no).

That "rule" can be easily overlooked and the net result is an overall slowdown, and this could be simply explained as this: C* needs to read more and more data (and it will all be read from one node only) to satisfy your query, keeping busy the coordinator, and slowing down the entire cluster. Data grow usually means slow query response, and after a certain threshold the infamous read timeout error.

That being told, it would be interesting to see if your DISK usage is "normal" or something is wrong. Give it a shot with dstat -lrvn to monitor your servers.

A final tip: depending on how many fields you are querying with SELECT * and on the amount of retrieved data, being served by an SSD may be not a big deal because you won't exploit the IOPS of your SSDs. In such cases, preferring an ordinary HDD could lower the costs of the solution, and you wouldn't incur into any penalty.

Upvotes: 1

Related Questions