CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-dynamodb
incnnu
incnnu

Reputation: 193

Export DynamoDB table to S3 automatically

The scenario is the following: I have a lambda function that does an http request to get the data of today and the last 365 days and stores them in DynamoDB. The function is triggered every day at 8am, so the most recent data is always saved in the DynamoDB table.

Now my goal is to export the DynamoDB table to a S3 file automatically on an everyday basis as well, so I'm able to use services like QuickSight, Athena, Forecast on the data.

If possible and easily implementable, I'd like to only have one S3 file that gets added with the most recent data of the day, because an extra file everyday seems kinda pricey. If that's not possible, an extra file everyday would also be fine.

What's the best way to go about doing so without using CLI (because I'm not allowed to install programs to my laptop) and without using Lambda (because I wouldn't know how to write a function for that without any tutorials)?

Upvotes: 2

Views: 7325

Answers (2)

elsyr
elsyr

Reputation: 796

DynamoDB recently released a new, native feature to export your table's data to an S3 bucket. It supports exporting into DynamoDB JSON and Amazon Ion - see the documentation on how to use it at:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataExport.html

This will enable you to run whatever analytics tools you'd like (Athena, etc.) on the data exported in S3.

Upvotes: 2

Chris Williams
Chris Williams

Reputation: 35146

Take a look at DataPipeline. This is a use case and most of the configuration is simple.

It will also not require any knowledge of Lambda and can be automated.

More info: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBPipeline.html

Upvotes: 1

Related Questions