CODEWITHSUNDEEP
javapaginationamazon-dynamodb
amy2017
amy2017

Reputation: 3

Pagination in DynamoDB when using QuerySpec.withMaxPageSize()

I want to use QuerySpec.withMaxPageSize(int limit) to implement pagination. I use it like this:

QuerySpec querySpec = new QuerySpec()
        .withKeyConditionExpression("id = :v_id")
        .withValueMap(new ValueMap()
            .withString(":v_id", "1234"))
        .withScanIndexForward(false)
        .withMaxPageSize(Integer.parseInt(5));
ItemCollection<QueryOutcome>items = table.query(querySpec);
int startPage = 0;
for (Page<Item, QueryOutcome> page : items.pages()) {
    System.out.println("Page: " + ++startPage);

    Iterator<Item> iterator = page.iterator();
    while (iterator.hasNext()) {
        System.out.println(iterator.next());
    }
}

Then the results is correct: every page has 5 items.

BUT when I use the QuerySpec like this:

QuerySpec querySpec = new QuerySpec()
    .withKeyConditionExpression("id = :v_id")
    .withFilterExpression("eventType = :v_eventType")
    .withValueMap(new ValueMap()
         .withString(":v_id", "1234")
         .withString(":v_eventType", "event"))
    .withScanIndexForward(false)
    .withMaxPageSize(Integer.parseInt(5));

THEN the results is totally wrong:

Page: 1
{ Item: {id=1234, timestamp=1489480336243, eventType=starRating} }
{ Item: {id=1234, timestamp=1489476430129, eventType=starRating, }
{ Item: {id=1234, timestamp=1489460649642, eventType=starRating, }
Page: 2
{ Item: {id=1234, timestamp=1489137400014, eventType=starRating} }
Page: 3
{ Item: {id=1234, timestamp=1489137294383, eventType=starRating} }
Page: 4
{ Item: {id=1234, timestamp=1489137046331, eventType=starRating} }
Page: 5
{ Item: {id=1234, timestamp=1489137030983, eventType=starRating} }
{ Item: {id=1234, timestamp=1489137022138, eventType=starRating} }
Page: 6
{ Item: {id=1234, timestamp=1489136930427, eventType=starRating} }
Page: 7
{ Item: {id=1234, timestamp=1489136782467, eventType=starRating} }
{ Item: {id=1234, timestamp=1489136765162, eventType=starRating} }
{ Item: {id=1234, timestamp=1489136669765, eventType=starRating} }
Page: 8
{ Item: {id=1234, timestamp=1489136478857, eventType=starRating} }
{ Item: {id=1234, timestamp=1489136453453, eventType=starRating} }
{ Item: {id=1234, timestamp=1489136435202, eventType=starRating} }
{ Item: {id=1234, timestamp=1489136384098, eventType=starRating} }
Page: 9
{ Item: {id=1234, timestamp=1489136253163, eventType=starRating} }

Every page has different items. It seems like when I add the filter expression, there will be this problem. But I don't know why. Does anybody knows about this?

Upvotes: 0

Views: 3533

Answers (1)

xtx
xtx

Reputation: 4456

Here is an excerpt from documentation that gives explanation why you are getting different number of rows per page:

In a response, DynamoDB returns all the matching results within the scope of the Limit value. For example, if you issue a Query or a Scan request with a Limit value of 6 and without a filter expression, DynamoDB returns the first six items in the table that match the specified key conditions in the request (or just the first six items in the case of a Scan with no filter). If you also supply a FilterExpression value, DynamoDB will return the items in the first six that also match the filter requirements (the number of results returned will be less than or equal to 6).

So basically when you use FilterExpression along with Limit in Query, DynamoDB reads up to Limit records from table and then filters out those that do not match your FilterExpression.

UPD: In order to get an exact number of records per each page, you would have to use a bit more complex approach. After you get the first query result, if the subset returned contains less than Limit items, save records into a buffer and continue reading until the total number of items becomes >= Limit. If the number of records in your buffer exceeds Limit, throw away exceeding records. That's how you get the first page with exact size of Limit items (5 in your case).

To get the second page, you could run a new query providing KeyConditionExpression like id = :v_id and #timestamp < :timestamp, and use timestamp value of the last record of the previous page for :timestamp. Again, you would have to read multiple times until the total number of items in buffer becomes >= your preferred page size. Use the same approach to get all subsequent pages.

Upvotes: 3

Related Questions