CODEWITHSUNDEEP
rubyredis
dsp_099
dsp_099

Reputation: 6121

Is there a better way of counting a large number of keys in Redis?

If I do something like

number_of_keys = redis_instance.keys('foo*').size

However there are 10,000 keys for this request, is there a way to speed up the process since I merely want to count them, and not return a whole batch of them first and then run the ruby .size method on them to count them up?

The problem is that returning 10,000 keys is a little slow and wasteful considering that I don't actually need them - all I need is to see how many there are.

EDIT: I should specify that redis_instance is an instance of redis class.

Upvotes: 2

Views: 694

Answers (2)

Eli
Eli

Reputation: 39009

So, before saying anything else, per the documentation:

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using sets.

So, if you can throw whatever it is you need to count into a hash and do an HLEN on that without things getting too messy, that would be way preferable.

Warning aside, you should 100% do this as a Lua script. Your bottleneck here without a doubt will be network throughput, and Lua solves this for you entirely. Lua will run your entire script server side, so you'll grab all the keys and count them all on your Redis box without any network transfer, and then you'll only send back the count. The script would be simple:

local all_keys = redis.call('KEYS', ARGV[1])
return #all_keys

Which you'd then call with just 'foo*' as the argument. In pure Redis, it would be (untested):

EVAL "your_script.lua" 0 'foo*'

I haven't used Redis in Ruby before, so you'll need to translate that into Ruby.

Upvotes: 3

anon
anon

Reputation:

In this situation, it would be better to use a counter to store a starting count and then increment the counter whenever a matching key is added. Regardless of which counting trick you find to be fast, it still relies on scanning your key space and looking for matches. Using INCR on a key (e.g. INCR starts_with_foo) will be quicker and, ultimately, more scalable.

Upvotes: 1

Related Questions