CODEWITHSUNDEEP
mongodbnosqlamazon-dynamodbtransitive-closure-table
Fahad Alrashed
Fahad Alrashed

Reputation: 1502

Exceeding NoSQL Item Size limit while preserving read/write performance

I need a way to exceed the NoSQL document item size limitations while preserving read/write performance.

All NoSQL implementations have document item size limits (e.g. AWS DynamoDB is 400KB, MongoDB is 16MB). Moreover, the larger the document item size, the slower the retrieval of the document becomes. Read, write, update and delete performance is important in my implementation.

I'm already using a NoSQL database and the documents I work with have a tree structure with four levels (The document describes vector drawings of varying sizes and complexity). Each node of the tree is a component that has an ID and a body that refers to the below components. Reading and updates can be performed for the full tree or parts of the tree.

I've considered switching to a Graph database but they are optimized to querying relationships in graphs. What I need a simple and high performance way to store an expanding tree documents rather than handling graphs.

I'm thinking that I need an implementation of Closure Table structure in NoSQL to split the tree nodes into separate components and - thus - breaking the document size barrier. On the client side, a ORM (object-relational mapper) with lazy-loading will retrieve the document tree nodes as necessary.

My questions are:

If you need such a functionality, upvote this so that more exports would answer this question.

Upvotes: 2

Views: 1016

Answers (1)

Lukas Liesis
Lukas Liesis

Reputation: 26403

DynamoDB does not have penalty of speed for size, yet networking will take extra time, that's it. Also if you have such a big file and trying to push it to DynamoDB, that's the design issue to begin with. Just store big file in S3 and store meta data in DynamoDB.

You are paying for write & read units by KBs with DynamoDB, why waste money on big data when S3 is so cheap for it and has the size limit is 5TB instead of 400KB. Also if your app generates those big JSON structures, you could save $ overtime by using S3 storage classes automation, which would move unused JSON to "colder" storage class.

Upvotes: 1

Related Questions