CODEWITHSUNDEEP
javapaginationamazon-dynamodb
Sindhu
Sindhu

Reputation: 2582

Dynamo Db query using maxPageSize and maxResultSize ,slowing down the query

I am using dynamo db pagination , Based on the AWS documentation :

--> maxResultSize is the maximum number of resources to be retrieved in this query, including all the resources in all pages to be retrieved.

--> maxPageSize is the maximum number of resources to be retrieved in a single page; it is used for pagination purposes.

Surprisingly , DynamoDb query is significantly faster when I do not set any page size (1) ,which should not be the case according to documentation . I am assuming dynamo db returns all the results by default ,if we do not specify pageSize .

querySpec.withExclusiveStartKey(lastAccessedRecord); (1)


querySpec.withMaxPageSize(30);
            querySpec.withMaxResultSize(100);
querySpec.withExclusiveStartKey(lastAccessedRecord); (2)

Upvotes: 1

Views: 4869

Answers (2)

Việt Đỗ
Việt Đỗ

Reputation: 11

ItemCollection: It indeed uses lazy loading. From AWS documentation:

Each call to {@code Iterator.next} on an Iterator returned from this Iterable results in exactly one call to DynamoDB to retrieve a single page of results.

On the other hand, the paging in DynamoDB is based on the size of the package (limit 1MB size). Each page returned will be 1MB max (the number of items will vary). By setting maxPageSize, DynamoDB would have to do more computation to fulfill your request instead of just returning matching items based on package size.

Upvotes: 1

Sindhu
Sindhu

Reputation: 2582

Root cause of this issue : ItemColletions is returning pages with 0 size as well .We need to break out of the loop as soon as there is page with size 0 to avoid iteration over unnecessary pages .

ItemCollections use some kind of lazy loading (only fetch results from DynamoDb when we traverse over pages )

"A collection of Item's. An ItemCollection object maintains a cursor pointing to its current pages of data. Initially the cursor is positioned before the first page. The next method moves the cursor to the next row"

QuerySpec spec = new QuerySpec().withKeyConditionExpression("Id = :v_id")
            .withValueMap(new ValueMap().withString(":v_id", replyId)).withMaxPageSize(1);

        ItemCollection<QueryOutcome> items = table.query(spec);

        System.out.println("\nfindRepliesForAThreadSpecifyOptionalLimit results:");

        // Process each page of results
        int pageNum = 0;
        for (Page<Item, QueryOutcome> page : items.pages()) {

if(page.size==0)   ](fix )
break;             ]
            System.out.println("\nPage: " + ++pageNum);

            // Process each item on the current page
            Iterator<Item> item = page.iterator();
            while (item.hasNext()) {
                System.out.println(item.next().toJSONPretty());
            }
        }
    }

Upvotes: 0

Related Questions