CODEWITHSUNDEEP
paginationamazon-dynamodb
Michael Gallego
Michael Gallego

Reputation: 1784

DynamoDB requires all fields for ExclusiveStartKey on GSI, why?

I'm trying to implement a cursor-based pagination using DynamoDB (definitely not easy to do pagination in DynamoDB...) using a query request using ExclusiveStartKey.

My table index is made of an "id", while I have a GSI on "owner" (partition key) and "created_at" (range key).

I can easily retrieve the 10 first records using a query request, by specifying the GSI index and using the "owner" property.

However, on next requests, the ExclusiveStartKey only works if I specify the THREE elements from both indices (so "id", "owner" AND "created_at").

While I understand for "id", and "owner" as those are both partitioned key and are needed to "locate" the record, I don't see why DynamoDb requires me to specify the "created_at". This is annoying because this means that the consumer must not only submit the "id" as cursor, but also the "created_at".

As DynamoDb could find the record using the "id" (which is guarantees unique), why do I need to specify this created_at?

Thanks

Upvotes: 7

Views: 5604

Answers (2)

broadbear
broadbear

Reputation: 674

In my testing, querying a gsi on a table resulted in a last evaluated key with all the item properties (essentially gsi key + table key). I needed to add all elements of the last evaluated key to the next request as exclusive start key to get the next page. If I excluded any elements of the last evaluated key in the next request, I received an exclusive start key error.

The following request:

aws dynamodb query --table-name MyTable --index-name MyIndex --key-condition-expression "R = :type" --expression-attribute-values '{\":type\":{\"S\":\"Blah\"}}' --exclusive-start-key '{\"I\":{\"S\":\"9999\"},\"R\":{\"S\":\"Blah\"},\"S\":{\"S\":\"Bluh_999\"},\"P\":{\"S\":\"Blah_9999~Sth\"}}' --limit 1

Resulted in the following response:

{
    "Items": [
        {
            "I": {
                "S": "9999"
            },
            "R": {
                "S": "Blah"
            },
            "S": {
                "S": "Bluh_999"
            },
            "P": {
                "S": "Blah_9999~Sth"
            }
        }
    ],
    "Count": 1,
    "ScannedCount": 1,
    "LastEvaluatedKey": {
        "I": {
            "S": "9999"
        },
        "R": {
            "S": "Blah"
        },
        "S": {
            "S": "Bluh_999"
        },
        "P": {
            "S": "Blah_9999~Sth"
        }
    }
}

If I left off some elements of the last evaluated key, for example (same request as above minus the table partition/sort keys):

aws dynamodb query --table-name MyTable --index-name MyIndex --key-condition-expression "R = :type" --expression-attribute-values '{\":type\":{\"S\":\"Blah\"}}' --exclusive-start-key '{\"I\":{\"S\":\"9999\"},\"R\":{\"S\":\"Blah\"}}' --limit 1

I get the following error:

An error occurred (ValidationException) when calling the Query operation: The provided starting key is invalid

Upvotes: 1

Alexander Patrikalakis
Alexander Patrikalakis

Reputation: 5205

GSI primary keys are not necessarily unique. Base table keys are necessary to answer the question, "For a given owner and creation date, up to which id did I read in this page?". Put another way, you could have multiple items with the same owner and creation date.

Upvotes: 2

Related Questions