CODEWITHSUNDEEP
javaamazon-web-servicesrecordamazon-dynamodbamazon-kinesis
luca
luca

Reputation: 3328

Data from Amazon Kinesis to DynamoDB

I have to pass my data from Kinesis to DynamoDB. The problem is that not all the data arrive when my software is running. So when I restart my application a lot of records are added into DynamoDB. Now I don't know why the records don't arrived all within run time, but this is a big problem. The monitoring of resources is good, i tried both InitialPositionInStream.LATEST and InitialPositionInStream.TRIM_HORIZON without any change. Is it happened at someone?Is there a solution? There seems to be a buffer that save record. I used git code of aws. Maybe be it a network problem?

Upvotes: 1

Views: 5820

Answers (1)

Chris Riddell
Chris Riddell

Reputation: 1024

Records in Amazon Kinesis are saved for 24 hours.

If you use TRIM_HORIZON, you would start at the oldest record (i.e. up to 24 hours old) and traverse forward from there. LATEST means you are only interested in records received since you start your new application.

I suggest you use the Amazon Kinesis Client Library if you are not already, which will manage the checkpointing of records using a DynamoDB table that it creates for itself to keep sequence. That way, when you stop and restart your application, it will save and resume from the last successfully processed record (This is still in a receive-at-least-once model).

Upvotes: 5

Related Questions