Reputation: 131152
I am building a PUB/SUB system backed by Redis.
I have one publisher and tons of subscribers. The subscribers are not that reliable, they can lose connection at any time and need to be able to "recover" from a lost connection.
There is a twist though, I would like my backlog capped at some number, meaning that a faulty subscriber should be able to recover only up to N messages.
The trivial solution is:
RPUSH list message
PUBLISH channel encoded
If a consumer needs to re-establish:
PSUBSCRIBE
atomicallyUp to here we are all good.
My big question is, what if I want the backlog list to be capped at N items?
Is there any way I can keep an ever increasing index AND a capped backlog in the list?
Upvotes: 6
Views: 1572
Reputation: 3569
How about this? To publish a message, do
LPUSH list message
LTRIM list 0 N
INCR global_index
PUBLISH channel global_index
When receiving a message over pub/sub and when starting up, clients will need to compare their latest index (this could also be kept in redis, or somewhere else) with global_index and read min(global_index - my_index, N) messages from the list to catch up (basically LRANGE list 0 (global_index - my_index)
).
Upvotes: 6