CODEWITHSUNDEEP
javaamazon-dynamodb
Lagani
Lagani

Reputation: 107

Getting all rows from an Amazon DynamoDB table

I need to get all rows from my DynamoDB table. There are approximately 500k records and every time I call scan method from Java SDK DynamoDBMapper I get different number of rows. So I tried to use scanPage method and use lastEvaluationKey from ScanResultPage object for pagination (because I think query result set size limit is 1MB) thinking it will work. But I get same result. I did something like this:

        List<String> ids = new ArrayList<>();
        DynamoDBScanExpression dynamoDBScanExpression = new DynamoDBScanExpression();
        do {
            ScanResultPage<EntityObject> scanResultPage = dynamoDBMapper.scanPage(EntityObject.class,
                    dynamoDBScanExpression);
            scanResultPage.getResults().forEach(result -> ids.add(result.getId()));
            log.info("Last evaluated key: {}", scanResultPage.getLastEvaluatedKey());
            dynamoDBScanExpression.setExclusiveStartKey(scanResultPage.getLastEvaluatedKey());
        } while (dynamoDBScanExpression.getExclusiveStartKey() != null);

What can I do to actually get all rows?

Upvotes: 2

Views: 2573

Answers (1)

smac2020
smac2020

Reputation: 10704

You are using a very old API and this API is not best practice. Amazon recommends the AWS SDK for Java V2.

For your use case, take a look at Retrieving paginated results using the AWS SDK for Java 2.x. This topic talks about how to handle AWS operations that return paginated results when the response object is too large to return in a single response. Details here:

Retrieving paginated results using the AWS SDK for Java 2.x

HEre is an example that demonstrates how to use auto pagination with the asynchronous DynamoDB client

https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/dynamodbasync/src/main/java/com/example/dynamodbasync/AsyncPagination.java

In the AWS SDK for Java 1.0, the response contained a token you had to use to retrieve the next page of results. New in the AWS SDK for Java 2.x are autopagination methods that make multiple service calls to get the next page of results for you.

Upvotes: 3

Related Questions