CODEWITHSUNDEEP
amazon-web-servicesamazon-dynamodb
Hoser
Hoser

Reputation: 5044

Options for paging and sorting a DynamoDB result set?

I'm starting on a new project and am going to be using DynamoDB as the main data source. A lot of what it does works perfectly for the needs, with a couple exceptions.

Those are the sorting and paginating needs of the UI. Users can sort the data by anywhere from 8-10 different columns, and a result set of 20-30k+ rows should be paginated over.

From what I can tell of DynamoDB, the only way to do sorting by all those columns would be to expose that many sort keys through a variety of additional indexes, and that seems like a misuse of those concepts. If I'm not going to sort the data with DynamoDb queries, I can't paginate there either.

So my question is, what's the quickest way once I have the data to paginate & sort? Should I move the result set into Aurora and then sort & page with SQL? I've thought about exporting to S3 and then utilizing something like Athena to page & sort, but that tool really seems to be geared to much larger datasets than this. What are other options?

Upvotes: 1

Views: 1708

Answers (2)

Charles
Charles

Reputation: 23793

Sorting, pagination and returning 20-30K records are not Dynamo's strong suit...

Why not just store the data in Aurora in the first place?

Depending on the data, Elasticsearch may be a better choice. Might even look at Redshift.

EDIT
If you've not seen this before... PIE Theorem

Upvotes: 1

Jason Wadsworth
Jason Wadsworth

Reputation: 8887

One option is to duplicate the data and store it once for each sorting option, with each version of the record having different data in the sort key. If you are okay with eventual consistency that might be a little more delayed you can accomplish this by having a lambda that reads from a DynamoDB stream and insert/update/delete the sorted records as the main records are inserted/updated/deleted.

Upvotes: 1

Related Questions