CODEWITHSUNDEEP
redisqueue
Mittenchops
Mittenchops

Reputation: 19664

Code implementation of the Redis "Pattern: Reliable queue"

The excellent redis documentation lists a Reliable queue pattern as a good candidate/example for the RPOPLPUSH function.

I understand "reliable queue" to be something with delivery patterns like Amazon SQS FIFO exactly once pattern.

Specifically, you have some N processes feeding into a queue, and some M workers working from the queue. What does this actually look like as an implementation?

I would venture something like:

Make the feeder process populating the work queue.

# feeder1

import redis
import datetime
import time

r = redis.Redis(host='localhost', port=6379, db=0)
while True:
  now = datetime.datetime.now()
  value_to_work_on = "f1:{}".format(now.second) 
  r.push('workqueue', value_to_work_on)
  time.sleep(1)

Make another

# f2
import redis
import datetime
import time

r = redis.Redis(host='localhost', port=6379, db=0)
while True:
  now = datetime.datetime.now()
  value_to_work_on = "f2:{}".format(now.second) 
  r.push('workqueue', value_to_work_on)
  time.sleep(1)

Now make the workers

# worker1
import redis

r = redis.Redis(host='localhost', port=6379, db=0)

def do_work(x):
  print(x)
  return True

while True:
  todo = r.rpoplpush("workqueue" "donequeue")
  if do_work(todo):
    print("success")
  else:
    r.push("workqueue", todo)

# worker2 is exactly the same, just running elsewhere.

My questions are:

  1. Is this generally what they mean in the documentation? If not, can you provide a fix as an answer?
  2. This seems still incomplete and not really reliable. For example, should there be alternative lists for error vs complete queues? One for every possible error state? What happens if your Redis goes down during processing?

Upvotes: 4

Views: 2783

Answers (1)

Mittenchops
Mittenchops

Reputation: 19664

As @rainhacker pointed out in comments, it is now recommended to use Redis Streams for this instead of the recipe described in "Pattern: Reliable Queue"

Upvotes: 4

Related Questions