CODEWITHSUNDEEP
amazon-web-servicesaws-lambdamqttamazon-sqsaws-iot
Esben von Buchwald
Esben von Buchwald

Reputation: 3049

AWS Lambda - how to identify duplicate messages

Since several of the triggers for AWS Lambda can only guarantee message delivery "at least once" (SQS and IoT with QoS=1), I wonder what's the best way to identify a duplicate message and ignore it.

I can see that I currently get several duplicate messages, triggering my lambdas twice, causing noise and invalid data as a consequence.

In my client, I solve it by just storing a list of message IDs that I've processed, but in the Lambdas, I have nowhere to store a state.

Of course I could maintain a DB table of processed message IDs but it seems like overkill to me (and probably adds extra billed runtime to the lambdas). A simple key/value store service in memory would be enough.

What other solutions are you guys using?

Upvotes: 3

Views: 2606

Answers (1)

macbutch
macbutch

Reputation: 3291

I know you don't want to use a DB but dynamodb can work well for this kind of thing. If you have something you can use as a good partition key then it will still be quite performant. It will still add a very small amount of time to your lambda run time and, of course, you will be charged for your dynamodb capacity & data. I use this successfully to discard duplicate messages.

The other thing that might be worth looking into would be elasticache which has memcached and redis versions. This would be faster - if performance is a particular focus - but is not persistent like DynamoDB.

Upvotes: 2

Related Questions