CODEWITHSUNDEEP
redis
Nick Long
Nick Long

Reputation: 1958

Redis Update Sorted Set on Key Expire

I have a Redis server with a set of key value pairs and a sorted set providing an index of the keys of those key value pairs.

The key value pairs can enter a "completed" state and at this point they will need to be deleted after 1 hour. This can be simply achieved by setting an expiry on the keys, but clearing them from the sorted set appears to be more problematic.

I could have a process that scans the sets and updates them from time to time, but it would be nice to have something a bit cleaner.

I don't think there is a way to trigger something like a stored procedure on expiry or set an expiry on the sorted set values.

Anything I've missed, or is manual the only way?

Upvotes: 2

Views: 2554

Answers (1)

Tug Grall
Tug Grall

Reputation: 3510

You are correct you cannot "put an expiry" on the sorted set value itself.

But you can work with the main key and capture the event when the expiration occurs. You have at least 2 ways to achieve this:

Key Space Notification

Using Key Space Notification, you can capture the EXPIRE event that will send a PUBLISH message that you can then consume.

Let me explain the basic flow:

Configure Notifications

CONFIG SET notify-keyspace-events Ex
  • E : events happening on keys
  • x : capture expired events

Now your database will publish an event on the __key*__:* channel.

So you can build a service that listen to these event update the set (directly or indirectly):

psubscribe __key*__:*

If you have an application that set the following value and expiration

set foo bar EX 5

You should receive the following message

1) "pmessage"
2) "__key*__:*"
3) "__keyevent@0__:expired"
4) "foo"

Redis Gears

Using Redis Gears, you capture the same event (it is based on notification too), but it is easier to write code directly in your Redis Database.

You can create a Gears as follow: (this is a Python script, I am using RedisInsight to deploy it to Redis)

def process(x):
    execute('LPUSH', 'expired:keys', x['value']['key']);

# Capture an expiration event and adds it to 'expired:events' stream
cap = GB('KeysReader')
cap.foreach(lambda x:
            execute('XADD', 'expired:events', '*', 'key', x['key']))
cap.register(prefix='*',
             mode='sync',
             eventTypes=['expired'],
             readValue=False)

# Consume new messages from expiration streams and process them somehow
proc = GB('StreamReader')
proc.foreach(process)
proc.register(prefix='expired:*',
              batch=100,
              duration=1, 
              trimStream = False)

Look at the section started by cap = GB('KeysReader')

  • This will listen to any key expiration prefix='*' & eventTypes=['expired']
  • In case of expiration it will add a message to the 'expired:events' Redis Stream using the XADD command
  • Then look at the function proc = GB('StreamReader') that will process the streams
  • each time a new message is in the stream it will call the process() function.

You can add your logic to update the sorted set in this function. In my example I have just added the expired key to a list.

Let me diverge a little bit from your initial question.

It looks like you are using Sorted Set to create some form of indexing for your data.

If this is the case, you should look at RediSearch, another Redis module that allow you to index Hash fields and then use the index to do some advanced queries and aggregation.

With RediSearch you do not need to add any code to manage the index, it is done automatically by the database, and you can query on the fields.

I am inviting you to look at:

Sorry if this is not the reason you are using Sorted Set, but I think it is worth checking as thus could simplify a lot your application code if you do manage index manually today.

Upvotes: 4

Related Questions