Redis dealing with queuing management mechanism

Question

I might be with the wrong mindset when approaching a queuing system with Redis, that's why I need your guys help for this.

So, I have this rather simple queue, responsible for stacking emails onto one collection and then retrieving the stack's available emails and dispatching it.

The system is built in NodeJS, so I'm using node-redis lib for that.

The queue (stack) is supposed to be constantly available, with one point adding (pushing) new emails on top of it, and with the other end, popping the first pushed item.

For this approach, I might be thinking too Javascript'ish, and with what I have found at Redis documentation, it might not be fitting, so here I come to you guys to help me acquiring the right mindset when talking Redis language.

Using a simple example, in JSON, here's my queue stack:

queue = [
 {
  _id: 5a05eec08a7e66eb10ad6361,
  email: "some html content in here",
  domain: "domain.com"
 },
 {
  _id: 5a05eb785710017b7d7a0243,
  email: "some more html content here",
  domain: "domain.com"
 },
 ...
]

And by looking at Redis Documentation I have found that I could push each email onto a stack by doing something like this:

HMSET queue:5a05eec08a7e66eb10ad6361 email "some html content in here" domain "domain.com"
HMSET queue:5a05eb785710017b7d7a0243 email "some more html content here" domain "domain.com"

And I could explicitly retrieve one 'collection' like this:

HMGET queue:5a05eec08a7e66eb10ad6361 email
HMGET queue:5a05eec08a7e66eb10ad6361 domain
HMGET queue:5a05eb785710017b7d7a0243 email 
HMGET queue:5a05eb785710017b7d7a0243 domain

Until this point, everything is pretty vanilla. But the problem is when it comes down to one queuing system, one really must use the PUSH and POP functionality the language/Database give you.

I was indeed able to find the PUSH and POP mechanisms Redis gives away, but I could only use it when dealing with single dimensioned KEYs.

So, in my case, instead of popping a single item like:

RPOP email 

I really need something like this:

RPOP queue //see the abstraction here? calling the stack and not a single item?

With in turn would return me - and in the same time remove - the last item at this queue collection.

So,

RPOP queue //or whatever other command I couldn't find

Should give me

 {
  _id: 5a05eec08a7e66eb10ad6361,
  email: "some html content in here",
  domain: "domain.com"
 }

... and then, with another

RPOP queue

return to me

{
  _id: 5a05eb785710017b7d7a0243,
  email: "some more html content here",
  domain: "domain.com"
 }

And so one, until it renders this "queue" empty.

I hope I was clear with exposing the mindset I have, as well as the problem itself.

I cannot iterate in the code by using the HMGET mechanism, for as any queue, while one end might be popping items from the bottom, the other end might be stacking new items on top of it. So, keeping via coding some 'reindexing' mechanism would be a little too much of a hack.

To this point, I'm starting to believe that Redis does not offer me the necessary tools to provide me support for this "multi dimensioned" stack approach. Which is fine imho, Redis might not be the answer for what I need.

But I'm aware that I might currently be with the wrong mindset, which probably Redis provides an elegant way to deal with that, but from a completely different approach. I coudn't just make it work with RPUSH and LPOP, for as I have already mentioned, it only offers me a way of dealing with simple single dimensional KEY VALUEs.

In case Redis does not provide some support for this, I already have implemented one workaround in the NodeJS end-point. But you know, more code means more potential bugs.

Peace

Answer 1

You don't need to break your queue's contents into discrete Hashes, unless you intend to access their subelements. If I you understand correctly, what you need is the classic LPUSH/RPOP list queue pattern. The members in that list can be anything, but in your case the simplest would be to just store the raw serialized JSON of each element, e.g.:

LPUSH queue '{_id: 5a05eec08a7e66eb10ad6361, email: "some html content in here", domain: "domain.com"}'