CODEWITHSUNDEEP
pythondjangoamazon-web-servicesbotoamazon-dynamodb
Sudip Kafle
Sudip Kafle

Reputation: 4391

Pagination with Dynamodb in web application

I have around 5000+ videos on my database and I have created a pages http://mysite.com/videos to list all the videos. Now I am implementing pagination so that only 20 videos are listed in each page. e.g.

http://mysite.com/videos?page=1 showing first 20 videos, http://mysite.com/videos?page=2 showing next 20 videos.

I have a problem choosing what is the best method to implement pagination. I thought of using table.scan() each time a new page is executed and then selecting only required based on some logic with Python code. But that seems to be quite expensive.

I am using Python / Django with boto library.

Upvotes: 0

Views: 4195

Answers (3)

Francis Shanahan
Francis Shanahan

Reputation: 2063

Limit is not what you think. Here's what I suggest:

Using DynamoDBMapper issue a 

numRows = mapper.count(<SomeClass>.class, scanExpression)

to efficiently get the number of rows in your table.

Then run a 

PaginatedScanList<FeedEntry> result = mapper.scan(<SomeClass>.class, scanExpression);

to get the List - the key here is PaginatedScanList is lazy-loaded. Be careful not to do a .size() on the result as this will load all the rows. Just use .get() to only load the rows you need.

Iterate over the paginatedScanList using 

offset = startPage * pageSize
ArrayList<SomeClass> list = new ArrayList<SomeClass>()
for (i = 0 ... pageSize) 
list.add(result.get( offset + i))

Check for out-of-bounds etc. Hope that helps.

Upvotes: 1

Dani C.
Dani C.

Reputation: 920

In Dynamo you can execute queries by setting a limit. From the documentation at:

http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html

you can read:

ExclusiveStartKey

The primary key of the first item that this operation will evaluate. Use the value that was returned for LastEvaluatedKey in the previous operation.

The data type for ExclusiveStartKey must be String, Number or Binary. No set data types are allowed.

Type: String to AttributeValue object map

Required: No

And

Limit

The maximum number of items to evaluate (not necessarily the number of matching items). If Amazon DynamoDB processes the number of items up to the limit while processing the results, it stops the operation and returns the matching values up to that point, and a LastEvaluatedKey to apply in a subsequent operation, so that you can pick up where you left off. Also, if the processed data set size exceeds 1 MB before Amazon DynamoDB reaches this limit, it stops the operation and returns the matching values up to the limit, and a LastEvaluatedKey to apply in a subsequent operation to continue the operation. For more information see Query and Scan in the Amazon DynamoDB Developer Guide.

Type: Number

Required: No

You don't provide any info about how the keys of your table are structured. However the approach would be Query the table for the elements matching your key (and range key if suitable), with the limit set to 20. In the result, you will receive a "LastEvaluatedKey", that you will have to use for the next query, again with the limit set to 20.

Upvotes: 6

tavi
tavi

Reputation: 680

Here are some of options:

  1. You can pre-load all video objects when the application starts up and then do in-memory pagination the way you want. 5000+ objects shouldn't be a big deal.
  2. You can fetch first page and then asynchronously fetch the rest (via scan) and then again do pagination in-memory.
  3. You can create an index table that would store an entry for each page containing the id-s of each video and then to fetch the video of a page you do to calls: 3.1 Fetch the page by page id (simple get operation). This will contain the list of video ids that should be on that page 3.2 Fetch all the videos from 3.1 by doing a multi-get operation

For a similar use case, we loaded all metadata via Javascript objects and do pagination and sorting from there and the end result for user is just nice (fast and responsive). As well, we're using the trick of fetching 1st page and then the whole content again (as DynamoDB doesn't support cursor at the moment)

Upvotes: 2

Related Questions