What happens when a lambda dies?

Question

I am new to AWS so I am not sure what the behavior is when the following situation occurs.

Let's say I have a Kinesis stream with JSON data (and let's say every couple of min a few thousand messages gets inserted).

Now there is a Lambda function that gets invoked everytime a new msg is inserted into the Kinesis which reads the msg and does some processing before inserting into Redshift.

So what happens if there is some error and the Lambda function crashes while doing the processing and takes a few minutes or even a couple of hours(i don't know if that's even possible) to come back up. Will it continue reading the Kinesis from the last unread message or will it read from the latest inserted messages (as that is the invoking event).

Thanks in advance.

Answer 1

Lambda function crashes while doing the processing

This is possible.

and takes a few minutes or even a couple of hours(i don't know if that's even possible) to come back up.

This is not exactly possible.

A Lambda function is only allowed to run until it returns a response, throws an error, or the timeout timer fires, whichever comes first. It would never be a couple of hours.

Lambda will create a new container every time the function is invoked, unless it already has one standing by for you or you are hitting a concurrency limit (typically 1000+).

However... for Kinesis streams, what happens is a bit different because of the need for in-order processing.

Poll-based (or pull model) event sources that are stream-based: These consist of Kinesis Data Streams or DynamoDB. When a Lambda function invocation fails, AWS Lambda attempts to process the erring batch of records until the time the data expires, which can be up to seven days.

The exception is treated as blocking, and AWS Lambda will not read any new records from the shard until the failed batch of records either expires or is processed successfully. This ensures that AWS Lambda processes the stream events in order.

https://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html

So your Lambda function throwing an exception or running past its timeout will simply cause the Lambda service to destroy the container immediately and create a new one immediately and then retry the invocation with the exact same data again until the data expires (as dictated by Kinesis config).

The delay would typically be no longer than your timeout, or the time it takes for the exception to occur, plus some number of milliseconds (up to a few seconds, for a cold start). The timeout is individually configurable on your Lambda function itself, up to 15 minutes (but this max is probably much too long).

It's potentially important to remember a somewhat hidden detail here -- there is a system that is part of the Lambda service that is reading your Kinesis stream and then telling another part of the Lambda service to invoke your function, with the batch of records. The Lambda service (not your Lambda function) is checking the stream by pulling data -- the stream is not technically pushing data to Lambda. DynamoDB streams and SQS work similarly -- Lambda pulls data, and handles retries by re-invoking the function. The other service is not responsible for pushing data.