CODEWITHSUNDEEP
design-patternsrabbitmq
yakka
yakka

Reputation: 613

Using MQ to send data from multiple clients to a central server

I have a bunch of nodes producing data, and I want to send all this data to be held by a central server. I am planning on using RabbitMQ to do this. All of the messages will be of a similar type, but it will matter to me which client it came from for how I store it.

My question is, what is the suggested pattern to do this? Should I have a single queue or multiple queues for each client? And if all the messages are coming out of a single queue, how can I keep track of where they come from?

Upvotes: 0

Views: 352

Answers (1)

robthewolf
robthewolf

Reputation: 7624

RabbitMQ works with exchange as well as queues. The producers can all send to the same exchange and they will be queued and the read by the client. The question is what are you trying to achieve?

There are options here:

  1. You can have all the producers write to one exchange and have one queue that receives these messages. Set the queue up to be durable and not autodelete. That way from outset it will collect all the messages that are sent to the broker. Your server will read from the queue using a single consumer, and process the messages accordingly. In this method each client must add a field to the message (the same field name for all messages from all clients) called "client name", for example. This field will give you the name of the client that sent the message so that when the server processes it, it will know who sent it.
  2. Alternatively, have the same process for sending the messages to a broker but set the broker up as a topic exchange. Send each message with a different routing key. Set up one queue for each client (which will have its own unique routing key), again each queue should be initialized at the start and be durable and autodelete=no, so all messages will be collected. Each queue will then receive only the messages from its own client. You will need a consumer set up for each individual queue. Which will have an over head, but will mean that messages are now processed in parallel. You will not need to check the message to know the client as each queue will have its own routing key and therefore you know that each consumer is only reading messages from specific clients.

There are positives and negatives with each. If you know how many clients (and their names) you will have and this is unlikely to change then (2) might be better. If you want something dynamic then you will need to use (1) as it is much easier rather than writing a new consumer each time you have a new client. If you want the parallelism of (2) but the simplicity of (1) then use a work queue set up for the consumer (server) side. http://www.rabbitmq.com/tutorials/tutorial-two-python.html Here you will have multiple consumers reading from the same queue to divide up the work.

Upvotes: 1

Related Questions