CODEWITHSUNDEEP
mongodbamazon-web-serviceselasticsearchamazon-s3amazon-dynamodb
mohitshah9920
mohitshah9920

Reputation: 31

DynamoDB vs ElasticSearch vs S3 - which service to use for superfast get/put 10-20MB files?

I have backend that recieves, stores and serves 10-20 MB json files. Which service should I use for superfast put and get (I cannot break the file in smaller chunks)? I dont have to run queries on these files just get them, store them and supply them instantly. The service should scale to tens of thousands of files easily. Ideally I should be able to put the file in 1-2 seconds and retrieve it in the same time.

I feel s3 is the best option and elastic search the second best option. Dyanmodb doesnt allow such object size. What should I use? Also, is there any other service? Mongodb is a possible solution but i dont see that on AWS, so something quick to setup would be great.

Thanks

Upvotes: 3

Views: 3990

Answers (1)

qkhanhpro
qkhanhpro

Reputation: 5220

I don't think you should go for Dynamo or ES for this kind of operation.

After all, what you want is to store and serve it, not going into the file's content which both Dynamo and ES would waste time to do.

My suggestion is to use AWS Lambda + S3 to optimize for cost S3 does have some small downtime after putting till the file is available though ( It get bigger, minutes even, when you have millions of object in a bucket )

If downtime is important for your operation and total throughput at any given moment is not too huge, You can create a server ( preferably EC2) that serves as a temporary file stash. It will

  • Receive your file
  • Try to upload it to S3
  • If the file is requested before it's available on S3, serve the file on disk
  • If the file is successfully uploaded to S3, serve the S3 url, delete the file on disk

Upvotes: 3

Related Questions