CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-dynamodbaws-elasticsearchaws-documentdb
geshu
geshu

Reputation: 21

Database suggestion for large unstructured datasets to integrate with elasticsearch

A scenario where we have millions of records saved in database, currently I was using dynamodb for saving metadata(and also do write, update and delete operations on objects), S3 for storing files(eg: files can be images, where its associated metadata is stored in dynamoDb) and elasticsearch for indexing and searching. But due to dynamodb limit of 400kb for a row(a single object), it was not sufficient for data to be saved. I thought about saving for an object in different versions in dynamodb itself, but it would be too complicated. So I was thinking for replacement of dynamodb with some better storage:

  1. AWS DocumentDb
  2. S3 for saving metadata also, along with object files

So which one is better option among both in your opinion and why, which is also cost effective. (Also easy to sync with elasticsearch, but this ES syncing is not much issue as somehow it is possible for both) If you have any other better suggestions than these two you can also tell me those.

Upvotes: 1

Views: 314

Answers (1)

Chris Williams
Chris Williams

Reputation: 35238

I would suggest looking at DocumentDB over Amazon S3 based on your use case for the following reasons:

  • Pricing of storing the data would be $0.023 for standard and $0.0125 for infrequent access per GB per month (whereas Document DB is $0.10per GB-month), depending on your size this could add up greatly. If you use IA be aware that your costs for retrieval could add up greatly.
  • Whilst you would not directly get the data down you would use either Athena or S3 Select to filter. Depending on the data size being queried it would take from a few seconds to possibly minutes (not the milliseconds you requested).

For unstructured data storage in S3 and the querying technologies around it are more targeted at a data lake used for analysis. Whereas DocumentDB is more driven for performance within live applications (it is a MongoDB compatible data store after all).

Upvotes: 1

Related Questions