CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-sqsaws-sqs-fifo
D3V
D3V

Reputation: 108

SQS FIFO to trigger lambda only after 10 messages in the queue

I will explain our use-case and our current approach (it might be inefficient) and the issue we have with our implementation.

Use case

Implementation

  1. All records from the MQTT topic will call a lambda which will process it in order to write it to the appropriate FIFO SQS queue
  2. The records on the SQS will trigger another lambda which in turn will call the dynamoDB ONCE to check if any of these record needs to be further processed in order to save it to dynamoDB

Problem

From step 2 on the implementation, we are aware that the batch limit is 10 for a FIFO SQS. But because for every record we have we need to make a GET request on dynamoDB we want to make the GET request every 10 records, not every 1 record in the queue. Currently, every new record in the SQS will fire the lambda which will make a GET request.

Is there any way we can "wait" or "ensure" we have the batch_limit full BEFORE triggering the lambda ?

I will illustrate it further with some dummy code/logic

dynamoDB = [{sensorID: 1, enabled: true}, {sensorID: 2, enabled: false}]

mqttData = [
     {sensorID: 1, data: "on"}, {sensorID: 1, data: "on"}, 
     {sensorID: 1, data: "off"}, {sensorID: 1, data: "on"}, 
     {sensorID: 1, data: "off"}, {sensorID: 2, data: "on"}, 
     {sensorID: 2, data: "on"}, {sensorID: 2, data: "off"}, 
     {sensorID: 2, data: "on"}, {sensorID: 2, data: "off"},
   ]

We have 10 records on the MQTT topic, 5 records from sensor1 and 5 records from sensor2. If the lambda processes each record separately it needs to make 10 dynamoDB requests to get the config for each sensor, whereas if we could process all 10 records in one go in the lambda, we would extract there are 2 unique sensors (1 & 2) and fetch the config for those 2 sensors and simply use that logic for the remaining records.

Thank you for your time.

Upvotes: 1

Views: 477

Answers (1)

Hakan Bakacak
Hakan Bakacak

Reputation: 31

I don't know the optimal solution but maybe you can set Batch window to 10 seconds.

Also there is another trick about lambda, if you write data to memory and the lambda doesn't die, memory is usable for next execution of lambda so you can save lastDynameDbCheckDate at the beggining of the function, you can reach it next execution.

if(now() - lastDynameDbCheckDate >10 sec)
    checkDynamoDb()
    lastDynameDbCheckDate = now()

In this way you request less to dynamodb

Does AWS Lambda reset memory on each invocation?

Upvotes: 0

Related Questions