CODEWITHSUNDEEP
rabbitmqmessage-queueamqp
thien
thien

Reputation: 406

RabbitMQ: Checking for queue size from the publisher

I was surprised to read on the rabbitmq docs that having very large queue sizes has a negative effect on queue performance. I have job dispatcher that can potentially enqueue 30k+ messages for job consumers to take up, and was noticing poor performance even after increasing the number of consumers. I then wrote code from the publishing side to check the size of the queue, and if its too large, sleep for a while until the queue size is smaller.

I found this to improve performance, but am questioning the logic behind this. It just seems smelly that the publisher has to be aware of such details of the queueing system that it has to rate limit itself, it kinda breaks the abstraction, doesn't it? Is this normal for all message queue systems? Are there alternative solutions, either with RabbitMQ or other libraries?

Upvotes: 0

Views: 475

Answers (1)

theMayer
theMayer

Reputation: 16167

You are correct that your publisher should not have to worry about the queue size. However, the programmer should:

  1. Ensure you have sufficient hardware and/or horizontal scale of your cluster to handle the message volume. 30k messages should not be a difficult thing for a single broker machine, assuming they are appropriately sized.

  2. Ensure your messages are no larger than about 100kb. If they contain large binary objects, you'll want to save those objects off to a data store and only publish a pointer to those resources in your message.

  3. RabbitMQ has flow control features to deal with overzealous publishers. I'm not sure what your settings are, but you might be able to tweak them to invoke this. Read more: https://www.rabbitmq.com/flow-control.html

Keep in mind that the ideal job queue size is zero, so your system should be designed to drive queue length as small as possible. This means processing messages at approximately the same rate they are produced.

Upvotes: 2

Related Questions