CODEWITHSUNDEEP
node.jsamazon-web-servicesaws-lambdaamazon-kinesis
VinuBibin
VinuBibin

Reputation: 785

How kinesis keep the offset and push the record again when an event fails in lambda

I am new to AWS lambda and Kinesis. Please help with the following question

I have a kinesis stream as a source to lambda and the target is again kinesis. I have following queries. The system doesnt want to lose a record.

if any of the records fails the processing in lambda, How it again pull into the lambda? How it keep the unprocessed records ? How kinesis track the offset to process the next record?

Please update.

Upvotes: 0

Views: 1272

Answers (1)

s.hesse
s.hesse

Reputation: 2060

From the AWS Lambda docs about using Lambda with Kinesis:

If your function returns an error, Lambda retries the batch until processing succeeds or the data expires. Until the issue is resolved, no data in the shard is processed. To avoid stalled shards and potential data loss, make sure to handle and record processing errors in your code.

In this context, also consider the Retention Period of Kinesis:

The retention period is the length of time that data records are accessible after they are added to the stream. A stream’s retention period is set to a default of 24 hours after creation. You can increase the retention period up to 168 hours (7 days)

As mentioned in the first quote, AWS will drop the event after the retention period is due. This means for you:

a) Take care that your Lambda function handles errors correctly.

b) If it's important to keep all records, also store them in a persistent storage, e.g. DynamoDB.

In addition to that, you should read about duplicate Lambda executions as well. There is a great blog post available explaining how you can achieve an idempotent implementation. And read here on another StackOverflow question & answer.

Upvotes: 1

Related Questions