CODEWITHSUNDEEP
pythonpython-2.7redis
samfr
samfr

Reputation: 666

Notification of key expiration in redis python

I would like to be notified when a volatile key expires in my redis store. The redis website provides some description of how this might be achieved in http://redis.io/topics/notifications, but I am wondering if it can be done using the python redis api.

After setting:notify-keyspace-events Ex in my redis.conf file

and running this as a test:

import redis
import config

client = redis.StrictRedis.from_url(config.REDIS_URI) 
client.set_response_callback('EXPIRE',callback)
client.set('a', 1)
client.expire('a',5)

callback() only gets called when client.expire('a',5) gets called, but not five seconds later as expected

Upvotes: 26

Views: 23136

Answers (1)

Jan Vlcinsky
Jan Vlcinsky

Reputation: 44102

The surprise (no expiration events seen when time to live for a key reaches zero) is not bound to Python, but rather to the way, Redis is expiring keys.

Redis doc on Timing of expired events

Timing of expired events

Keys with a time to live associated are expired by Redis in two ways:

  • When the key is accessed by a command and is found to be expired.
  • Via a background system that looks for expired keys in background, incrementally, in order to be able to also collect keys that are never accessed.

The expired events are generated when a key is accessed and is found to be expired by one of the above systems, as a result there are no guarantees that the Redis server will be able to generate the expired event at the time the key time to live reaches the value of zero.

If no command targets the key constantly, and there are many keys with a TTL associated, there can be a significant delay between the time the key time to live drops to zero, and the time the expired event is generated.

Basically expired events are generated when the Redis server deletes the key and not when the time to live theoretically reaches the value of zero.

Small test on console

when Redis running ($ sudo service redis-server start)

I started one console and have subscribed:

$ redis-cli
PSUBSCRIBE "__key*__:*"

Then, in another console:

$ redis-cli
> config set notify-keyspace-events AKE

what shall subscribe to all kinds of events

Then I continued with experiments in this second console:

> set aaa aaa
> del aaa
> set aaa ex 5
> get aaa

All the activities were seen in subscribed console. Only the key expiration was sometime few seconds delayed, sometime came just in time.

Note alse, there are subtle differences in messages, one message __keyevent@0__:expire another __keyevent@0__:expired.

Sample listener spy.py

import redis
import time

r = redis.StrictRedis()
pubsub = r.pubsub()
pubsub.psubscribe("*")
for msg in pubsub.listen():
    print time.time(), msg

This code registers to all existing channels in default redis and prints whatever gets published.

Run it:

$ python spy.py

and in another console try to set a key with an expiration. You will see all the events.

For following redis-cli input.

$ redis-cli
127.0.0.1:6379> set a aha
OK
127.0.0.1:6379> set b bebe ex 3
OK
127.0.0.1:6379> set b bebe ex 3
OK

we get spy output:

1401548400.27 {'pattern': None, 'type': 'psubscribe', 'channel': '*', 'data': 1L}
1401548428.36 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyspace@0__:a', 'data': 'set'}
1401548428.36 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyevent@0__:set', 'data': 'a'}
1401548436.8 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyspace@0__:b', 'data': 'set'}
1401548436.8 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyevent@0__:set', 'data': 'b'}
1401548436.8 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyspace@0__:b', 'data': 'expire'}
1401548436.8 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyevent@0__:expire', 'data': 'b'}
1401548439.82 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyspace@0__:b', 'data': 'expired'}
1401548439.82 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyevent@0__:expired', 'data': 'b'}
1401548484.46 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyspace@0__:b', 'data': 'set'}
1401548484.46 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyevent@0__:set', 'data': 'b'}
1401548484.46 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyspace@0__:b', 'data': 'expire'}
1401548484.46 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyevent@0__:expire', 'data': 'b'}
1401548487.51 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyspace@0__:b', 'data': 'expired'}
1401548487.51 {'pattern': '*', 'type': 'pmessage', 'channel': '__keyevent@0__:expired', 'data': 'b'}

Upvotes: 38

Related Questions