Reputation: 8626
I have three SQS FIFO queues where each one has a data projection listener daemon in EC2 instance as docker pods (SQL Server, PostgreSQL, Elastic Search, etc.)
All queues have the same settings as below (Dead-Letter Queues to setup later).
Queue Type: FIFO
Messages Delayed: 0
Content-Based Deduplication: Enabled
Default Visibility Timeout: 30 seconds
Message Retention Period: 14 days
Maximum Message Size: 256 KB
This is all part of an Event Sourcing architecture I am designing using DynamoDB Stream => Lambda SQS Router => SQS FIFO Queues (due to SNS not supporting FIFO queues as subscribers)
Content-Based Deduplication
is enabled to avoid duplicate messages in the queue since an error is always possible in the Lambda Router for any of the queues.
Now, I also have set the MessageGroupId
for each message to the AggregateId to group them but don't really understand how that is utilized by the consumer side;
I have only one consumer per SQS queue at the moment but what if I want to scale consumers. Is an application concern to make sure multiple consumers will not process messages from the same MessageGroupId; - which is unacceptable since using FIFO queues is due to order retention of events in the system!
Upvotes: 3
Views: 5618
Reputation: 269081
If a message has been received from a FIFO Amazon SQS queue and it is still invisible ("in-flight"), then SQS will not provide another message with the same MessageGroupId
.
Therefore, multiple consumers on the same queue will receive messages with a different MessageGroupId
and message order within a given MessageGroupId
will be retained.
The important thing here is to use a different MessageGroupId
where you wish to retain order, but do not use the same MessageGroupId
for every message.
See: AWS SQS FIFO - How to get more than 10 messages at a time?
Upvotes: 10