CODEWITHSUNDEEP
redis
Prav
Prav

Reputation: 401

Iterating through keys in Redis

I have just started with Redis. My DB contains about 1 billion records. Using HKEYS * results in an out of memory error.

Is there a way to iterate through keys? Something like HKEYS * but with a limit n?

Edit:

I am now using a loop which matches a pattern

for c in '1234567890abcedf':
    r.keys(c + '*')

Upvotes: 12

Views: 28349

Answers (4)

Vaibhaw
Vaibhaw

Reputation: 646

Available since Redis 2.8.0 are the cursor based Redis iteration commands (SCAN, HSCAN etc) that let you iterate efficiently over billions of keys. For your specific case, the start using HSCAN instead of HKEYS/HGETALL. It is efficient, cheap on server resources and scales very well. You can even add a pattern to HSCAN unlike HKEYS.

e.g.

127.0.0.1:6379> HMSET hash0 key0 value0 key1 value1 entry0 data0 entry1 data1
OK
127.0.0.1:6379> HSCAN hash0 0 MATCH key* 
1) "0"
2) 1) "key0"
   2) "value0"
   3) "key1"
   4) "value1"
127.0.0.1:6379> HSCAN hash0 0 
1) "0"
2) 1) "key0"
   2) "value0"
   3) "key1"
   4) "value1"
   5) "entry0"
   6) "data0"
   7) "entry1"
   8) "data1"

Upvotes: 10

Lloyd Moore
Lloyd Moore

Reputation: 3197

Sorry, at the current time, year 2012, the simple answer is no, however, with lua scripting you could do it, although that is not direct redis in the strictest sense.

Upvotes: 0

Niels
Niels

Reputation: 1603

For iterating through keys:

SCAN cursor [MATCH pattern] [COUNT count]

http://redis.io/commands/scan

For iterating through the values of a hash

HSCAN key cursor [MATCH pattern] [COUNT count]

http://redis.io/commands/hscan

Upvotes: 0

rianjs
rianjs

Reputation: 7954

You can't iterate over redis keys directly, but you can accomplish something very similar by transactionally writing the key portion of your key-value pair to a sorted set at the same time you write your key-value pair.

Downstream, you would "iterate" over your keys by reading n keys from the sorted set, and then transactionally removing them from the sorted set at the same time as you remove the associated key-value pair.

I wrote up an example with some C# code here: http://rianjs.net/2014/04/how-to-iterate-over-redis-keys/

You could do this in any language that has a redis library that supports transactions.

Upvotes: 0

Related Questions