CODEWITHSUNDEEP
ubuntunetwork-programmingredis
Nyxynyx
Nyxynyx

Reputation: 63599

Retrieving large datasets from redis

An app on one server queries redis which runs on another server. The result data set is about 250k from a query zrangebyscore objects:locations -inf +inf which seems to take 40 seconds on the app server.

When the command is performed using redis-cli on the redis server or the app server, in both cases they also take around 40 seconds to complete as reported by redis-cli.

The redis server uses about 15% of CPU during the query.

Question: Is taking 40 seconds to retrieving 250k records considered slow? Is it possible to speed it up to take seconds?

Upvotes: 4

Views: 2110

Answers (1)

Didier Spezia
Didier Spezia

Reputation: 73206

First, it depends on the average size of the items.

On my system, retrieving 250K items of 10 bytes with a zrangebyscore takes only 113 ms. With 100 bytes items, it takes 228 ms. With 1 Kb items, it takes 4033 ms.

So except if your items are much bigger than this, I would say your 40 seconds response time is pretty bad.

Redis is not designed to work well with virtual memory. If Redis memory is swapped out, performance is generally catastrophic, so I would say your first action should be to avoid Redis memory to be swapped out.

I would try to dump the Redis database (with bgsave, to be sure that all pages are back in RAM), and then try the zrangebyscore again (and several time) to see if the response time is better.

Update:

Here is the command I use to build the content:

$ python -c 'for x in range(0,250000): print "ZADD objects:locations 0.1 %0115d" % (x)' | redis-cli | wc

Here is the command I use to time the query:

$ time redis-cli -h <hostname> zrangebyscore objects:locations -inf +inf >/dev/null

Upvotes: 3

Related Questions