CODEWITHSUNDEEP
javaamazon-dynamodbdynamodb-queriesaws-java-sdk-dynamodb
gerrytan
gerrytan

Reputation: 41123

Numeric limit & offset based pagination in DynamoDB (Java)

I want to implement numeric limit and offset based pagination in DynamoDB similar to Postgres.

My API look something like this: http://foo.bar/user?offset=50&limit=20.

What's the best way to do this in Java without risking OutOfMemoryError considering DynamoDB uses ExclusiveStartKey and LastEvaluatedKey to paginate?

EDIT:

Let's assume offset based pagination is a hard-requirement and I don't know the 'prior' page. My API contract have offset and limit query params as described above. I'm not looking for "don't do offset based pagination" answers.

Upvotes: 4

Views: 2360

Answers (1)

gerrytan
gerrytan

Reputation: 41123

After looking at how PaginatedList work in the sdk, it seems the most efficient way is to use ITERATION_ONLY pagination loading strategy in the config, and do something like this:

DynamoDBMapperConfig config = new DynamoDBMapperConfig.Builder()
                       .withPaginationLoadingStrategy(ITERATION_ONLY)
                       .build();
PaginatedQueryList<MyUser> users = dynamoDBMapper.query(MyUser.class, myQueryExpression, config);
// This iterator will fetch 1MB worth of data 'on-demand'
Iterator<MyUser> userIterator = users.iterator();
skip(iterator, offset);
List<MyUser> aPageOfUser = collect(iterator, limit);

Of course I'll incur the cost of implementing my own skip and collect method there. Is there a better / more-concise way to do this?

EDIT:

It seems I can utilise IteratorUtils from commons-collections to have the skip and collect for free:

Iterator<MyUser> userIterator = IteratorUtils.boundedIterator(users.iterator(), offset, limit);
List<MyUser> aPageOfUser = IteratorUtils.toList(userIterator);

Upvotes: 4

Related Questions