CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-sqs
Lee Netherton
Lee Netherton

Reputation: 22532

Pattern to guarantee processing of files in an SQS queue using Lambda

I have a very simple task to perform on many files as they are uploaded to an S3 bucket. The task can easily be performed by a simple Lambda function, and the resulting files written back to S3.

Connecting this Lambda function to an event triggered by S3 events is trivial, but I have some added complications:

  1. The files must be processed (if the Lambda function fails then the operation needs to be attempted again, until it succeeds, given a reasonable amount of retries)

  2. I need to know when all files have finished being processed

It seems to me that a putting a message queue in the system (SQS) seems to be a sensible option. That way, any failed messages will be retried after the Visibility timeout. Also I will be able to query the queue length to see if there are still operations in flight.

I can connect S3 events to an SQS queue. The problem is that I cannot directly trigger Lambda invocations from an SQS queue.

File --> S3 --> SQS --??--> Lambda ----> S3
                                    \
                                     `-> If successful delete message from SQS

I could use Kinesis to take messages from the queue and trigger lambda functions, but this seems a bit overkill? Likewise, I could have dedicated instances polling the queue and operating on it, but for such a simple function, I don't really want to have to run a cluster of instances.

Is there a good design pattern for this?

Upvotes: 3

Views: 1421

Answers (1)

Mircea
Mircea

Reputation: 10566

For S3 to Lambda directly, I think you're right that you need to worry about the lambda function not correctly working / crashing and the processing for that particular S3 object not taking place.

That being said, I would suggest doing S3 to Lambda directly coupled with a mechanism that allows you to detect and re-request processing of items that failed to process (i.e. a nanny). You could periodically schedule a job through Lambda (http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html) that looks for old items (older than a certain threshold) that were not processed. Most processing is going to be done on the direct (happy) path.

One way this could be done would be using SQS with delayed delivery and with lambda scheduled function. So you would have 

File --> S3 --> Lambda --> S3 (happy path)
             \-> SQS (delayed)
                  \-> Lambda(scheduled job) -> S3 (unhappy path)

In the scheduled job you could check if processing happened and just ack the message. The scheduled job can trigger the same Lambda job used for the normal processing. Also, don't forget to configure a dead letter queue for the SQS queue to catch things that fail processing over multiple retries.

Keep in mind that you could potentially bypass SQS completely if you have a way of figuring out what was processed and what was not process by inspecting the S3 bucket.

Also keep in mind that all of the above assumes that it okay if you process something that was already process (i.e. edge cases involving timeouts on lamda function, delay on the queue and sqs delivery)

Upvotes: 1

Related Questions