CODEWITHSUNDEEP
rabbitmqmessage-queueservicebusmasstransitmessagebroker
Jose Alonso Monge
Jose Alonso Monge

Reputation: 1034

How to organize queues in Masstransit/RabbitMQ?

I'd like to know best practices for consuming messages. I've read MassTransit docs and I've searching about this but I don't get to come to any conclusion.

I have one api (hosting a bus instance) that is publishing messages. These messages are varied because this api is not a microservice (messages for purchases, sales, etc).

How do I have to organize my consumers/queues?

  1. One process for queue type? For example one for purchases, other for sales, etc. this solution could involve having many processes and I'm not sure whether or not it is a good solution. What if I want diferent queues for purchases, like purchases.stock, purchases.suppliers, etc? Process number could increase considerably. I think this is a good option for scalability, but manage so many processes could be tricky.
  2. Multiple queues for process (grouping queues by domain)? For example one process having multiple consumers consuming purchases related messages and managing diferents queues, like purchases.stock, purchases.suppliers... This option makes more sense to me, but I'm not sure about it.

Upvotes: 6

Views: 1783

Answers (1)

Chris Patterson
Chris Patterson

Reputation: 33298

The key concept is to avoid sharing a single queue for different message types. There are exceptions, but keeping each message type on a separate queue prevents bottlenecks when one message type dominates the queue traffic.

As for processes, since MassTransit can have any number of receive endpoints per bus instance, keeping related business functions in a single process can help significantly with code management and deployment. The process boundaries can be useful for scaling, for instance, adding more processes/workers for handling status updates versus new orders (the prior may be 10x the latter in terms of message volume).

Another reason to separate is dependencies, a consumer that communicates with a legacy ERP system with a lot of bindings or coupling to external libraries/SDKs might warrant a separate process just to avoid memory issues due to the way some older libraries are created. These libraries may require more frequent process restarts, etc. and keeping them separate eliminates the need to restart an entire set of consumers for the one that causes issues over time.

These are just some general guidelines, but ones that we use all the time when determining which consumers to put in the same process.

Upvotes: 4

Related Questions