Ensuring Reliable Recovery of Missed Redis Sorted Sets in Event of Consumer App Downtime

Question

I have a system with two applications:

infraApp: Writes data to Redis every 2 minutes using sortedSet. Each sortedSet key is based on the current date and time in the format yyyy-MM-dd-HH:MM, for example, 2024-09-26-15:12.

processorrApp: Reads the sortedSet data from Redis, processes it by sending it to a queue, and then deletes the sortedSet. But, this processorApp is an executable. A cronjob keeps crashing and running this executable. processorrApp is a node.js app.

Problem: If the processorrApp goes down, infraApp continues to write data to Redis. When processorrApp is back online, I want it to consistently process the data it missed during its downtime and then continue processing from where it left off.

Constraints:

I need to ensure no data is lost and each sortedSet is processed exactly once. infraApp must not be affected by the downtime of processorrApp. Question: What is the best way to design this system to ensure consistent processing of missed data when the processorrApp is back online? Any specific Redis patterns or queue mechanisms that would be suitable for this scenario?

Additional Details: Redis is used for temporary data storage before processing. The date-time format used as the sortedSet key ensures unique keys every 2 minutes. I am open to using other tools or mechanisms if necessary.

Answer 1

You can use Scan command, which I would offer to use anyway, even if the processorApp didn't go down for safety, unless you have some heavy usage that you can afford the resources.

Instead of pulling base on current time exactly one SortedSet, iterate over your cache server and pull all the keys, then perform the action on the keys you pulled.

If you have different type of keys as well, or you have some keys you don't want to get rid of for any reason — use MATCH option.
If, let's say, one day back is safe enough for your needs, scan with MATCH=2024-09-26* then with MATCH=2024-09-25*, then you'll get all keys from today and yesterday.
You can also specify TYPE=SortedSet than you'll get only keys from type sorted set. \ Be aware — SCAN only return keys, not values. After aggregating your keys, use them to pull the values.