CODEWITHSUNDEEP
rabbitmqmqtt
user3684457
user3684457

Reputation: 39

rabbitmq & MQTT - architecture for a cluster of clients

I have several servers (for load) that listen to all the messages (MQTT) with wildcard, however I only need each message to be processed only once by one of the servers. How can I architect that?

Thanks guys!

Upvotes: 1

Views: 2034

Answers (2)

Adrian Hofman
Adrian Hofman

Reputation: 1451

It's practically impossible to guarantee that each message will be processed exactly once in this scenario.

A consumer could go down after it's processed the message, but before it's been acked, meaning that the message remains in the queue and can be processed again.

You can de-duplicate messages at the consumers - that is, the consumer can store the IDs of processed messages somewhere, so that it can check each new incoming message to see if it's been processed already. If it has, simply ack the message and don't process it again. As you have scaled out consumers, the de-duplication storage needs to be shared between all consumers to work (e.g., a database).

Even this measure won't strictly guarantee exactly once only processing - see here for more details: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-August/008272.html

Upvotes: 0

Dominik Obermaier
Dominik Obermaier

Reputation: 5770

This sounds like a use case for a "classic" message queue with many consumers. Please note, that root-wildcard subscribers are a huge antipattern in MQTT since slow consuming clients can slow down the whole system if no sophisticated backpressure handling is implemented.

I have seen people doing something similar what you want to achieve with the following architecture:

  1. A HiveMQ MQTT broker which is on the MQTT edge. This broker implements a custom plugin (with the open source plugin system) which forwards every MQTT message (with JMS, AMQP or HTTP) to another message broker like AWS SQS.
  2. The consumers just consume the message off the message queue.

This works very well in a cloud environment, especially since you don't have to care about scaling the message queue since e.g. SQS does this automagically for you. You can add more HiveMQ servers on the edge and the backend MQ also scales up.

The advantage of this solution is, that you can scale very well since all components are duplicatable, you can have more HiveMQs (in a cluster for example) if your MQTT load increases and you can scale your consumers up if more load is to be processed.

HiveMQ 3 (will be released in Q1 2015) will support your desired behaviour out of the box (distributing messages on a topic amongst subscribers with an only-sent-once semantic), so this works perfectly without any additional system. You have to wait for this out-of-the-box solution a few months, though.

Obligatory disclaimer: I'm involved developing HiveMQ, so as always: This answer may be biased.

Upvotes: 2

Related Questions