CODEWITHSUNDEEP
redis
requiem31
requiem31

Reputation: 177

Using large strings as Redis keys with a list of values

We have an Antlr parsing API which returns the given columns being accessed in a query. Only ~1/100 queries are unique so we are looking at using Redis as a caching layer to get a drastic speed increase. Some queries are thousands of lines long and take a full second or more to parse. Our estimated volume is in the hundreds of millions so we cannot afford to parse any duplicates.

Looking at Redis (and using python redis client), should I hash each query text with something like MD5 and use that as a key and use rpush to store the columns for that query as a list?

Or is hashing before in that manor a waste of time. Im also looking at Redis' own hashing functions like HMSET, but it does not look like there is a great way to store a list as a value for a key.

Upvotes: 1

Views: 1890

Answers (1)

Kevin Christopher Henry
Kevin Christopher Henry

Reputation: 48992

Your basic idea is a good one, but if you're just using this for caching there's little point in dealing with Redis lists. Those are used when you want to operate on the data within Redis itself (inserting new elements into the list, etc.). Instead you can just use regular GET and SET.

Specifically, use the hash as the key and some encoded form of the data (JSON, or whatever you like) as the value. It's possible you could skip the hashing step (Redis allows keys up to 512MB), but if queries are "thousands of lines long" that would eat up your cache memory and make serialization and transmission significantly slower.

Upvotes: 2

Related Questions