CODEWITHSUNDEEP
amazon-web-services
Hemant
Hemant

Reputation: 141

aws kinesis get-records returns empty array

I am playing around with Kinesis, and I tried a very simple example

I first put a sample record

aws kinesis put-records --records "Data=Test data - hemant,PartitionKey=20150421" --stream-name mystream

I get back

{
"FailedRecordCount": 0,
"Records": [
    {
        "SequenceNumber": "49549975503580743304507290523786194552567002802960728066",
        "ShardId": "shardId-000000000000"
    }
]}

So put appears to have worked. Now I am trying to retrive back this record, first, by getting the shard-iterator, and then calling the gert-record using the returned shard-iterator. The get-shard-iterator returns

aws kinesis get-shard-iterator --stream-name cakestream --shard-id 0 --shard-iterator-type LATEST
{
"ShardIterator": "AAAAAAAAAAEna1yL0ccbircK95wu6WRfN7LamlaXL5bZ1GzaFrUcSU8S74o4Pus59Z0XmdaMamdvz4tv3qKuPxpomz/Eeg671gVUKNHUDruAKyA4pjWRP37VI1K5w/kLqpBo49YsCKHMxcduaN6GdeCXL4QMSgvH9Aqi7leRuIr2T1w4MeqjhlcM1iz8icaWGlHfUVCbgtY="}

And now I try to get the records using that shard-iterator

aws kinesis get-records --shard-iterator "AAAAAAAAAAEna1yL0ccbircK95wu6WRfN7LamlaXL5bZ1GzaFrUcSU8S74o4Pus59Z0XmdaMamdvz4tv3qKuPxpomz/Eeg671gVUKNHUDruAKyA4pjWRP37VI1K5w/kLqpBo49YsCKHMxcduaN6GdeCXL4QMSgvH9Aqi7leRuIr2T1w4MeqjhlcM1iz8icaWGlHfUVCbgtY="{
"NextShardIterator": "AAAAAAAAAAE4lTq/jqanuj+xsULhl6QQeykzToObYDoaukearHkQfed/keYjgxzwfxkDXlBJBAOVLsk3pI9d0EwQWn5NmJ9poCL9M1wGDe2M42fgmp1EdK0WJGI1zG7TMi8m1bGQ6qDL05zf7gCtK5/xod6Vw/Gr98bsdQ8Ewp3U57FuHxZ29LUUbYp3AoN7CbUTD5rtqzU=",
"Records": []}

So, my question is why am I not getting back my data?

Upvotes: 14

Views: 7828

Answers (4)

Upendra
Upendra

Reputation: 91

Few things here...

  • It is not guaranteed that you would end up getting your records in the very first attempt. You will have to loop through the iterators(next iterator is returned with every getRecords() call) to eventually get all your data.

  • There are various ways to get the first iterator. You might want to use the correct one based on your use case

    Valid Values: AT_SEQUENCE_NUMBER | AFTER_SEQUENCE_NUMBER | TRIM_HORIZON | LATEST | AT_TIMESTAMP

Upvotes: 2

Teddy Payne
Teddy Payne

Reputation: 109

It seems there is stream name mismatch.

You have put your records in mystream stream but you are trying to fetch those records from cakestream stream.

Upvotes: 1

Karthikeyan Marudhachalam
Karthikeyan Marudhachalam

Reputation: 186

As Paedolos mentioned, LATEST shard iterator will return only the records put after the iterator is created.

If you would like to process all the records in the stream from beginning, you need to create TRIM_HORIZON shard iterator.

Upvotes: 2

Paedolos
Paedolos

Reputation: 280

The LATEST shard iterator will return all records put into the stream after it was created.

So, if you'd like to see your records, you'll have to put them after you create the iterator, and then request for get-records.

Upvotes: 8

Related Questions