CODEWITHSUNDEEP
architecturerabbitmqmessage-queueamqp
contactmatt
contactmatt

Reputation: 18600

Scaling slightly long running Consumer work - RabbitMQ

I'm architecting RabbitMQ into our solution, and I'm curious on how to handle processing and acknowledging messages efficiently, while still performing "real" work in the consumer code that can range from 5-10 seconds. (More work than the samples ever delve into).

enter image description here

Above is an example of what I'm aiming to process. A message in my twitter.tweet_cmd_q queue that has all the parameters needed in the message body, for the consumer to make the actual Twitter API request, and save those results to the DB.

However, I'm running into two problems here:

  1. I'll be processing thousands of records a minute - I can't possibly take 5-10 seconds before acknowledging that message. Is it "normal" for the consumer code to process all the needed work before acknowledging the message? (i.e. I could see ack'ing the message, and throwing the actual work to be done in another thread for processing. - though this would then require it's own form of "thread" management so that the system does undergo too much load).

  2. Would RPC calls benefit me at all here, in this situation that involves querying the data and saving it to the DB?

  3. Is the best way to deal with this scalability, just to create more worker instances for round robin handling?

Upvotes: 1

Views: 1677

Answers (2)

Luke Bakken
Luke Bakken

Reputation: 9627

Rob's answer is great, I just wanted to add to this:

Is it "normal" for the consumer code to process all the needed work before acknowledging the message?

If you acknowledge the message before processing it, and your consumer crashes or otherwise does not complete its task, that message is lost. That is the main reason to only ack when your work is complete. You can find the relevant documentation here: https://www.rabbitmq.com/confirms.html

Upvotes: 2

Rob Conklin
Rob Conklin

Reputation: 9446

  1. Yes, it is normal for the consumer code to do everything it needs with the message before ack'ing it. That's how you manage the message visibility to other workers.

  2. No, keep it simple, just do what you need to do in the worker.

  3. Sort of, it's not really round robin, just create more workers, and have them subscribe to the queue. Each worker will poll the queue, find messages, and execute on them.

Upvotes: 4

Related Questions