CODEWITHSUNDEEP
redis
WinterTime
WinterTime

Reputation: 183

Manage more than million results in REDIS

I am pretty new using REDIS but I already see some limitations or I am not sure what to use.

So the use case is that I am doing some batch computing based on the data from SQL, which run separate thread/GAE instance, for example 100k is the chunk size, then I need to somehow store this 100k data in REDIS and access it from different thread/GAE (but I don't really need whole SET/HASH/Value, I need a single one out of this 100k).

For example each of the value I wanna store as this:

{myData1: {a: 'a', b:'c'}, myData2:{a: 'a', b:'c'}, myData100000: {a: 'a', b:'c'}}

And I want to retrieve for example myData70000 values.

Key point is that, I need to use REDIS as (some sort of Map) collection where I can append data (so other threads can append another 100k into previous 100k etc., or overwrite single results) as well as fetch single key from the collection as example above in O(1) complexity.

I could easy create single key/value ref in REDIS like myData1: JSON.strigfy(data), but the computing can create approx 50 mil references which are then batched again for 1mil chunks (so the natural choice would be to have them as some sort of collection), are inserted to SQL and then purged.

What would be the natural way of doing such a complex things in REDIS?

Upvotes: 0

Views: 2511

Answers (1)

tabreaz
tabreaz

Reputation: 659

Redis hashes are very suitable for such processing it provides O(1) time complexity.

Also consider sharding your hashes for reducing total memory footprint. See the link below how instagram store such information (https://instagram-engineering.com/storing-hundreds-of-millions-of-simple-key-value-pairs-in-redis-1091ae80f74c)

Redis document this scenarios pretty well ( https://redis.io/topics/memory-optimization) Hope this will provide you enough information

Upvotes: 1

Related Questions