CODEWITHSUNDEEP
amazon-web-servicesnosqlamazon-dynamodbmicroservicesserverless
Franklin
Franklin

Reputation: 891

DynamoDB access pattern change

I am using DynamoDB as the database of choice in a very large and complex system.

This was done around 2017/18 and the application has been working and evolving ever since. The decision of using DynamoDB was based on the access patterns and scalability properties. Recently we are doing a face-lift of our UI and it drastically changes some of the access patterns we designed our database and model for.

We don't want to migrate out of DynamoDB so I am thinking of creating an auxiliary store in (SQL or similar) that replicates our DynamoDB table but let's us do all the needed queries of the data.

More details

The table that is changing holds many entities. The change is for one of this entities. It used to get accessed sorting by timestamp, and paging was done using pointers (infinite scroll in the UI).

The new design is great. Gives the user more power to massage the data, but changes drastically the access patterns we designed the model and table for.

We are not looking to change our database. This entity has 10+ attributes but one of them is a tree like structure, and JSON is perfect and DynamoDB as well. We also use heavily Streams and TTL for post processing and other stuff. Our whole application relies on this DynamoDB table to be the source of truth for this entity.

Some possible solutions, that I can think of, but I am not convinced.

  1. Create an Aurora Serverless Postgress DB that holds a view of our entity. It acts as a replica of our entity in DynamoDB. We can achieve this in application code by writing in DynamoDB and then in this SQL store. This will let us do all the queries with SQL (offset, limit, order by, etc). Like a durable cache.
  2. OpenSearch. Use an index in open search. Very similar approach to the previous one, except instead of having a SQL counter part we have an (Elastic Search) Open Search index that holds a subset of this entity to do all the operations.
  3. Add a bunch of indexes and do parallel querying in DynamoDB. Basically code all the possible sort, filter and paging logic in our application code.

Upvotes: 0

Views: 250

Answers (1)

hunterhacker
hunterhacker

Reputation: 7132

A fourth option is Rockset. It can scan what’s in DynanoDB and stay up to date with Streams, then provides a variety of ad hoc query capabilities against the data including text search features and analytics.

It’s serverless and does all the sync work for you, two big benefits. I’d try that first.

Upvotes: 1

Related Questions