CODEWITHSUNDEEP
cluster-computingrabbitmqscalabilityamqpspring-amqp
user2539645
user2539645

Reputation: 81

RabbitMQ queue design and scaling

In my application I have a queue that could potentially become very big. What if I discover that there's no more space on my machine? How can I split my queue on multiple machines? Maybe RabbitMQ philosophy is different and I should create multiple queues instead of one big queue..?

Best, Flavio

Upvotes: 6

Views: 6920

Answers (2)

pinepain
pinepain

Reputation: 12859

RabbimMQ provides clustering and high availability features right out of the box, you just have to configure them to your needs.

Actually, AMQP can hold messages of any size, but in most cases messages are just up to 32Kb in 99%, I guess. You can calculate estimate resource usage (min/max/avg) and make further decision do cluster or not to cluster.

Upvotes: 1

user2539645
user2539645

Reputation: 81

As you can read on RabbitMQ mailing list thread http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-scalability-design-question-td28323.html, the solution I came up with is to implement the competing consumers pattern (many consumers on a queue) that when detects a special message (with stop flag on) sends a STOP message to a topic exchange.

This stop message is received by a "master" consumer for that queue that starts polling a Zookeeper (via curator) until all children of a ceratain zkNode have been deleted (using Zookpeer as a registry of queue-consumers in this case). When all consumers have finished their stopping phase the "master" consumer does some task and re-enable the original queue consumers sending a RESTART message to the topic exchange (where each consumer is listening to with a dedicated queue).

I hope this could help (or "inspire") someone else..

Upvotes: 2

Related Questions