CODEWITHSUNDEEP
rabbitmqamqp
Steve Nesh
Steve Nesh

Reputation: 85

RabbitMQ - Avoiding duplicate messages at publisher side

I am using RabbitMQ to send notifications to the user. The user can read his queue at any time.

The problem I am facing is that the queue is filled with lots of notifications during the night, and when the user returns in the morning, he has to process these messages sequentially. A lot of these notifications are even duplicates.

I guess it would make sense to improve this at the publisher side. That is, before adding a new notification, we investigate if there are already pending notifications in the queue. If this is the case, we only queue a new notification if it is really a new notification, hence avoiding duplicates.

We might even go further and extend this by combining notifications: instead of simply queuing a new notification, we could replace the present notifications from the queue by a new one which holds the sum of these notifications and the new one (for example in an array of inner notifications).

Is this possible with AMQP/RabbitMQ?

Upvotes: 5

Views: 16085

Answers (2)

noxdafox
noxdafox

Reputation: 15030

This rabbitmq plugin has been written to tackle your issue.

You can enable de-duplication on a queue via setting its x-message-deduplication argument to true.

Then, your publishers will need to provide the x-deduplication-header message header with a value meaningful for de-duplication. The value could be a unique message ID or the MD5/SHA1 hash of the body for example.

Upvotes: 6

Gabriele Santomaggio
Gabriele Santomaggio

Reputation: 22682

No, by default you can't replace an existing message.

Upvotes: 3

Related Questions