CODEWITHSUNDEEP
redis
user1354934
user1354934

Reputation: 8841

How to get all keys/values from redis in order to insert them into SQL db?

I have a lot of analytics data that I'm adding to redis. I plan on incrementally moving the data out of redis and into my database.

I know I can use KEYS [the_key]:* to get all keys that match. For example, I can do that to get the following:

127.0.0.1:6379> KEYS c_Track:*
1) "c_Track:6c93a5c1-77e9-4c4a-9232-bf182713a02e"
2) "c_Track:2c9d99c2-af37-4de9-ac64-b48f339e97a9"
3) "c_Track:9e7fd190-86d9-4b4a-9a70-7bf4c7768eef"
4) "c_Track:7f2d2e98-7440-4fd7-a80a-2af309ab15a4"

Is there a recommended way to get these values easily? I can get the keys, but how can I get all the values as well? I can loop through the keys to get the values, but is there some one-shot method for doing this?

Also I know I shouldn't use keys, but this is just an example. Thanks

Thanks

Upvotes: 3

Views: 798

Answers (1)

Itamar Haber
Itamar Haber

Reputation: 49942

Also I know I shouldn't use keys

So don't. Use SCAN instead.

is there some one-shot method for doing this?

No, not as a core Redis command, but given the need this is fairly simple to achieve with a server-side Lua script. For example, assuming that your values are strings, you could do something like the following:

local cursor = tonumber(ARGV[1])
local pattern = ARGV[2]
local scan = redis.call('SCAN', cursor, 'MATCH', pattern)
for i, v in ipairs(scan[2]) do
  local val = redis.call('GET', v)
  scan[2][i] = { v, val }
end

return scan

Assuming that this script is saved under "scan.lua", you can run it as follows:

$ redis-cli SET foo bar
OK
$ redis-cli SET baz qaz
OK
$ redis-cli --eval scan.lua , 0 "*"
1) "0"
2) 1) 1) "baz"
      2) "qaz"
   2) 1) "foo"
      2) "bar"

To scan your entire keyspace, call the script with the returned cursor until it returns 0.

Notes:

1) If your keys are of different types, you should change the script accordingly (e.g. https://github.com/itamarhaber/redis-lua-scripts/blob/master/scanfetch.lua).

2) While this script goes against the common recommendation of generating key names inside a script, it is still safe to run as SCAN returns keys that are in the server's keyspace (whether single-instance or clustered).

Upvotes: 4

Related Questions