CODEWITHSUNDEEP
rubyredis
Stuart
Stuart

Reputation: 1134

Redis: Get all keys from queue which have a value equal to some criteria

I have a bunch of jobs I store on redis, each identified by a special key, with a hash as their value that contains some information. These jobs get picked up, computed, and once they are done, they have a success field that is set to true if the computation was finished successfully. I want to populate a list of all the keys that have the success key in their value hash set to true.

e.g. redis storage:

foo_key_1 => {bar_hash_key: bar_value, baz_hash_key: baz_value, success: true}
foo_key_2 => {bar_hash_key: bar_value, baz_hash_key: baz_value, success: false}

With the example above, I'd like a simple, efficient way of scanning through all keys in redis and their success field, and end up with [foo_key_1] as a result.

Currently my approach is as follows (in ruby pseudocode):

# redis is a connection to my redis server
all_keys = redis.keys                      # list of all keys in redis
for i in all_keys do                       # iterate through all keys in redis 
  if (redis.hgetall i)["success"] == true  # if that key has success = true attrib
    completed_keys.append(i)               # append that key to a new list
  end
end

The problem, as you might guess, is I have a lot of keys on redis, and this, albeit O(N) in performance gets fairly sluggish in terms of performance, due to its iterative nature. I have pored through the redis docs/commands but nothing leapt out at me as a potential candidate to solve this problem more efficiently, I am no redis guru, but it seems fairly straightforward, you put stuff in and take stuff out. Are there any possible vectorized operations I might have overlooked?

Many thanks.

Upvotes: 0

Views: 3588

Answers (2)

Kache
Kache

Reputation: 16667

To do this efficiently only using Redis, you would have to keep an updated, "Redis-queryable", SET or LIST of success/completed keys.

At a fundamental level, it sounds like what you really have here is a job queue (and a second job queue for the successful first jobs). Unless you specifically want to implement this on your own, have you looked into job queue implementations?

Upvotes: 1

Razvan Pavel
Razvan Pavel

Reputation: 351

You cannot do it without iterating through all keys, because redis supports only key fetching. You can reduce that complexity if you happen to know a pattern for the keys where you have to look, even in this case redis will fetch all the keys in background(with some optimizations, because it will fetch them in batches http://www.rubydoc.info/github/redis/redis-rb/Redis:scan). Something like:

$redis.scan_each(match: "foo*").to_a

You can find how to write your own pattern here https://redis.io/commands/KEYS

However, if you can control what values the keys take you can make them take a value with a little hint to help you in this search, like:

  • let's make each key with this form "foo#{your_number}:#{success_bool}"
  • when search use this pattern "foo*:1"

Upvotes: 2

Related Questions