How to automatically remove an expired key from a set in Redis?

Question

I have, say, 4 keys in a Redis database. The keys have an expiration of 10 seconds from now. I have added the keys to a set. When the keys expire, they are effectively not in the database anymore (get returns null value). However, the keys are still members of the set. The set will continue to exist until the keys are removed from the set, at which point it too effectively is gone from the database.

Is it possible to automatically remove a key from the set(s) to which it belongs when the key expires?

I am thinking of maintaining one set per key to maintain the sets which it belongs to (making a digraph set relationship between keys and sets), and then registering for key expiration events to remove the set members when necessary. This is a lot of overhead to do as a consumer of the db, rather than as some kind of background cleanup thread inside. And, such an approach would be "best-effort" to cleanup sets, as the consumer code which subscribes to the expiration events might crash, not get the notification, get backlogged, etc.

I may have been able to avoid any trickery by modeling the set of keys as fields in a hash set, but I wish to have different expiration TTLs per key in practice. If that is possible, then how so?

For example, here is my set of "foo" keys.

mine:0>set foo1 barA
OK

mine:0>set foo2 barB
OK

mine:0>set foo3 barC
OK

mine:0>set foo4 barD
OK

They can be added to a set.

mine:0>sadd foo foo1 foo2 foo3 foo4
4

mine:0>smembers foo
1) foo1
2) foo3
3) foo4
4) foo2

The keys can then expire...

mine:0>expire foo1 10
1

mine:0>expire foo2 10
1

mine:0>expire foo3 10
1

mine:0>expire foo4 10
1

mine:0>get foo1 
NULL

mine:0>get foo2
NULL

mine:0>get foo3
NULL

mine:0>get foo4
NULL

mine:0>get foo5
NULL

At this point the keys don't exist, but the set does, and it references the keys.

mine:0>smembers foo
1) foo1
2) foo3
3) foo4
4) foo2

Removing the keys explicitly will remove them from the set, and then make the set non-existent once all are removed.

mine:0>srem foo foo1
1

mine:0>srem foo foo2
1

mine:0>srem foo foo3
1

mine:0>srem foo foo4
1

mine:0>smembers foo
[nothing returned]

Answer 1

it's simple!

First, I wanna point out that data in redis do not have any reference relationship whatsoever! The foo1 as a key and foo1 as an element in the set is two toltally different things, different data in different memory address, which happens to store the same string "foo1".

Second, My way don't need to change you data model, just the way you query your set. When you query elements in the set, check if the key is also exist as a seperate key in redis, the logic is as below:

ArrayList getUnexpiredElementsInSet(String setKey){
            ArrayList elements = smembers(setKey);
            ArrayList resultArray = new ArrayList();
            for (element in elements) {
                if (exists(element)==false) {
                    srem(setKey, element);
                }else {
                    resultArray.add(element);
                }
            return resultArray;
          }

In this way when you want to query things in a set, you only get unexpired elements, and also the expired elements is cleaned.

Above is just pseudo java code to show you basic logic, you need to wrap this logic in a lua transaction. If you do not know redis's lua feature, just learn it! it's great and make redis super fast and powerful. If you want to exploit the real power of redis, lua is indispensable! Refer to Redis Lua

Some worry about performance penalty of scripts commands. I can assure you that's unnecessary. According to the official doc and what I experienced in practice, running scripts have no performance penalty at all. This is theoretically understandable in many ways:

1. Redis adopt Lua rather then any other language because Lua claims to be the fastest script language.
2. Every script can be stored on the server side and can be invoked by client's sending only the sha1 value of that script and its input args, so the I/O burden of long scripts don't exist.
3. When running script through sending sha1 of the stored scripts, the scripts is precompiled on redis server. So, in this way, running commands wrapped in scripts is essentially the same as running those commands in plain old form, except scripts save a lot of time-consuming I/O operations. No wonder it's super fast.