CODEWITHSUNDEEP
amazon-web-servicesamazon-kinesis
suman j
suman j

Reputation: 6960

AWS Kinesis read from past

How do we read from AWS Kinesis stream going back in time?
Using AWS Kinesis stream, one can send stream of events and the consumer application can read the events. Kinesis Stream worker fetches the records and passes them to IRecordProcessor#processRecords from the last check point.
However If I have a need to read the records going back in time, such as start processing records from 2 hours ago, how do I configure my kinesis worker to fetch me such records?

Upvotes: 3

Views: 2854

Answers (1)

itai ariel
itai ariel

Reputation: 569

You can start your kinesis consumer again (or a different one) with different settings regarding the Shard iterator. see here GetShardIterator The usual setting is LATEST or TRIM_HORIZON (oldest):

{
   "ShardId": "ShardId",
   "ShardIteratorType": "LATEST",
   "StreamName": "StreamName",
}

But you can change it to a specific time (from the last 24 hours)

{
   "ShardId": "ShardId",
   "ShardIteratorType": "AT_TIMESTAMP",
   "StreamName": "StreamName",
   "Timestamp": 2016-06-29T19:58:46.480-00:00
}

Keep in mind that usually the kinesis consumer saves its checkpoints in a dynamodb table, so if you are using the same kinesis application you need to delete those checkpoints first.

Upvotes: 4

Related Questions