CODEWITHSUNDEEP
amazon-web-servicesmodelnosqlamazon-dynamodb
user2741831
user2741831

Reputation: 2388

Model Post and Topic through DynamoDB

Heres the relation I'm trying to model in DynamoDB: My service contains posts and topics. A post may belong to multiple topics. A topic may have multiple posts. All posts have an interest value which would be adjusted based on a combination of likes and time since posted, interest measures the popularity of a post at the current moment. If a post gets too old, its interest value will be 0 and stay that way forever (archival).

The REST api end points work like this:

The problem now becomes, how do I create a relationship between topics and and posts in DynamoDB NoSql?

I thought about adding a list of copies of posts to tag entries in DynamboDB, where every tag has a list of both the newest and the most interesting Posts. One way I could do this is by creating a cloudwatch job that would run every 10 minutes and loop through every topic object, finding both the most interesting and newest entries and then replacing the old lists of the topic. Another job would also have to regularly update the "interest" value of every non archived post (keep in mind both likes and time have an effect on the interest value).

One problem with this is that a lot of posts in the Tag list would be out of date for 10 minutes in case the User makes a change or deletes the post. Likes will also not be properly tracked on the Tags post list. This could perhaps be solved with transactions, although dynamoDB is limited to 10 objects per transaction.
Another problem is that it would require the add-posts-to-tags job to load all the non archived posts into memory in order to manually sort them by both time and interest, split them up by tag and then adding the first N of both sets to the tag lists every 10 minutes.

I also had a another idea, by limiting the tags of a post that are allowed to 1, I could add the tag as a partition key, with the post-time as the sort key, and use a GSI to add Interest as a second sort key.
This does have several downsides though:

But it would have the advantage that there are no replications of the post objects aside from the GSI. It would also allow basically infinite paging of all posts by date instead of being limited to just the N newest posts.

So what is a good approach here? It seams both of my solutions have horrible dealbreakers. Is this just one of those problems that NoSQL simply can't solve?

Upvotes: 1

Views: 107

Answers (1)

Shachaf.Gortler
Shachaf.Gortler

Reputation: 5735

You are trying to model relational data using a non relational DB , to do this I would use 2 types of DB , I would store in dynamo the post information in your example it would be :

GET /posts/{id} 
POST /posts/
POST /likes/creates

For the topic related information I would use Elastic search (Amazon Elasticsearch Service)

GET /topics/{name} : the search index would stored the full topic info as well post id's that , and the relevant fields you want to search for (in your case update date to get the most recent posts)

what this will entail is background process (in dynamoDB this can be done via streams) that takes changes to the dynamoDB for new post's , update to like count etc.. and populates the search index.

Note: this can also be solved using graphDB but for scaling purposes better separate the source of the data (post's ) and the data relations (topic).

Upvotes: 1

Related Questions