CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-dynamodbamazon-kinesis-firehoseamazon-dynamodb-streams
Reyan Chougle
Reyan Chougle

Reputation: 5321

AWS - Identify data read by Lambda from DynamoDB Streams

I have a DynamoDB table that has TTL and DynamoDB Streams enabled/configured.

I want to implement a lambda function that will read the TTL deleted data from DynamoDB Streams and forward it to maybe Kinesis Firehose or S3 (this I need to decide which one is better considering cost).

Is there any flag/property that will help to identify that the TTL deleted record is already being read/processed by some lambda function? For instance, I have 10 records in DynamoDB Stream, a lambda function has read 5 records and did some processing on it (forwarding it to Kinesis Firehose or S3). So, I want to identify those 5 records which were processed.

Upvotes: 0

Views: 682

Answers (3)

mango
mango

Reputation: 568

To summarize: ask is to track the records which has already been TTL deleted. (I presume, you somehow wants to ensure not to reprocess them again)

Pretext:

  • As called out from @korgen, you would be able to verify userIdentity in stream record to identify the TTL deleted vs normal delte
  • You can relay on DDB on the fact that DDB stream contains unique events(not repeated or duplicate)
  • each stream record has a recordIdentifier(its a GUID) which helps track the location of record in stream. Alternatively you may use the HashKey + Range Key (+ record version, if present on DDB record) to build your own custom key

On top of my head, one solution that you can explore would work like this:

  1. While processing the TTL deleted record, persists in a secondary storage (uploading to s3).
  2. This way you can check this storage, if the record exist, then your lambda has already processed it.

Upvotes: 0

Marcin
Marcin

Reputation: 238209

Sadly there is no such "flag/property" which indicates that a given DynamoDb record is being processed already or not yet by a lambda function. You would need to design a custom solution for that.

One would would be to have other DynamoDb table, just to keep track of a current status of the deleted items. For example, when your lambda accepts a record it writes it in the table. Any other function or interested processes would have to check if the record exists or not in the table. The presence of the record would indicate that the item deleted is still being process.

Upvotes: 0

Korgen
Korgen

Reputation: 5399

you can use the userIdentity field in the stream record to identify which record came from DynamoDB's TTL deletion.

See the details here: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/time-to-live-ttl-streams.html

Upvotes: 2

Related Questions