CODEWITHSUNDEEP
amazon-web-servicesamazon-kinesis
Vyacheslav Yanuk
Vyacheslav Yanuk

Reputation: 13

Retention period kinesis

I developed application with Spark-streaming+Kinesis. I started my app at 2015-10-30 and it works. I saw Today (2015-11-02) I restarted the application, and it began to handle 2015-11-02. Why? Retention period? In docs

"You can increase the retention period up to 168 hours using the IncreaseRetentionPeriod operation"

I tried increase with command as describe in docs

aws kinesis increase-stream-retention-period --stream-name XXXX --retention-period-hours 160

But

usage: aws [options] [parameters] aws: error: argument operation: Invalid choice, valid choices are: add-tags-to-stream | create-stream delete-stream | describe-stream get-records | get-shard-iterator list-streams | list-tags-for-stream merge-shards | put-record put-records | remove-tags-from-stream split-shard | wait help

What should I do for increase retention period???

WBR, Slava

Upvotes: 1

Views: 775

Answers (2)

Mircea
Mircea

Reputation: 10566

you need to update the aws cli that you are using. you're probably using an older version that does not know about the newer command option.

@az3's answer is also worth looking into to understand why you pick up processing where you currently do. (he explains the behavior and I tend to agree :) )

Upvotes: 1

az3
az3

Reputation: 3629

  • I don't think it is related to retention period. If you are using Kinesis Client Library (KCL), you may be persisting the last worked position to DynamoDB. So if you stop and start your application, it will continue to work from the last known position.

http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-ddb.html

  • If you want to read all the previous data in a shard, you should update the checkpoint in DynamoDB to "TRIM_HORIZON".

http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-additional-considerations.html

  • When application sees TRIM_HORIZON, it will read the first data (approximately ~23 hours 55 minutes before) in Kinesis Stream. You can find more about retention here;

http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-extended-retention.html

Upvotes: 0

Related Questions