CODEWITHSUNDEEP
restasynchronousrabbitmqmessage-queue
ChrisM
ChrisM

Reputation: 2248

RabbitMQ temporary queues for status updates in asynchronous REST

I am designing a REST API which works according to the asynchronous design detailed here. I am using RabbitMQ to enqueue the initial requests - so the client makes a call, receives a 202 Accepted response, and the job is enqueued by the server. In order that clients can get status updates ('percent done') on tasks we have a secondary queue resource, just as in the linked article.

Given that each task has its own queue resource it seems we need one temporary RabbitMQ queue per task. I am wondering whether this is a wise design choice, even though I can't really see any other options. It seems unlikely to be very efficient, and I am uneasy about the possibility of having lots of temporary queues being created like this, especially as I cannot see a way to guarantee that they will all be cleaned up (despite RabbitMQ's auto-delete facility). Prior to RabbitMQ I was using SQS for this, and have painful experience of what can happen in this respect.

I note that a similar type of queue management will be already familiar to those using RabbitMQ in RPC style. Is there a possible alternative, however?

Upvotes: 0

Views: 2053

Answers (1)

pinepain
pinepain

Reputation: 12869

Firs of all, each queue utilize apr. 20k memory, so having a lot of them is up to you and your hardware. But in general, it smells. Really.

For status updates I see nothing wrong to use some key-value database, like redis or even memcache and update percent done there. Thus status check (as well as updating) will be damn fast, simple and lightweight.

Update:

I can suggest further architecture:

  1. Client POST task payload to some endpoint, say /tasks.
  2. Application generate unique task id (uuid aka guid is your friend here), published that task with it id to RabbitMQ queue and then return id to client.
  3. Workers (one or many) consume tasks from RabbitMQ and depends of processing step update Redis key which has task id with some value (step, percentage done, estimated time to receive result). So, it may be looks like SET task:{id} "<some valye>". When task completed by worker it MAY update Redis key with task result or store it somewhere else and then set Redis key represent task is finished.
  4. Client MAY time to time GET /tasks/{id} to receive task status or it result.
  5. When Application receive GET /tasks/{id} it return task status represented by Redis key (GET task:{id}). If key is not set (nil) then task is not yet taken by worker.

P.S.

RPC is something different from what you asked, but i would recommend to read this question for some details.

Upvotes: 5

Related Questions