CODEWITHSUNDEEP
goconcurrencyrabbitmq
Sam
Sam

Reputation: 499

Is it possible that multiple consumers of a Rabbitmq queue get the same message?

I am in charge maintaining a production software written in Golang which uses RabbitMq as its message queue.
Consider the following situation:

  1. A number of goroutines are publishing to a queue name logs.

  2. Another set goroutines read from the queue and write the messages to a MongoDB collection.

  3. Each publisher or consumer has its Own connection, and its own channel respectively, they are working in an infinite loop and never die. (The connections and channels are established when the program starts.)
  4. autoAck, exclusive and noWait are all set to false and prefetch is set to 20 with global set to false for all channels. All queues are durable with autoDelete, exclusive and noWait all set to false.

The basic assumption was that each message in the queue will be delivered to one and only one consumer, so each message would be inserted in the database exactly once.
The problem is that there are duplicate messages in the MongoDB collection.
I would like to know if it is possible that more than one consumer gets the same message causing them to insert duplicates?

Upvotes: 2

Views: 3386

Answers (1)

Olivier
Olivier

Reputation: 2691

The one case I could see with your setup where a message would be processed more than once is if one of the consumers has an issue at some point. The situation would follow such a scenario:

  1. Consumer gets a bunch of messages from the queue
  2. Consumer starts processing a message
  3. Consumer commits the message to mongodb
  4. either due to rabbitmq channel/connection issue, or other type of issue consumer side, the consumer never acknowledges the message
  5. the message as it hasn't been acknowledged is requeued at the top of the queue
  6. same message is processed again, causing the duplication

Such cases should show some errors in your consumers logs.

Upvotes: 2

Related Questions