CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-dms
T4puSD
T4puSD

Reputation: 142

How to configure AWS DMS to keep multiple full load files in the same s3 target destination?

I have a table running on AWS RDS. I want to use AWS DMS to export all the data on the table every week. Each week after the export I will truncate the table so every next phase the source table will have new data and I planned to perform the DMS task to safely offload the data from the RDS table.

I have configured an RDS source and S3 bucket as target to export data as CSV. The replication type is Full-Load only and it migrates existing data(No Ongoing replication).

But the problem I found is that DMS keeps dropping the old LOADXXXXXXX.csv file from the target s3 whenever I perform the reload-target operation on the DMS task next week.

How can I achieve my goal? How to configure AWS DMS to keep multiple full load files in the same s3 target destination?

Upvotes: 1

Views: 1852

Answers (2)

T4puSD
T4puSD

Reputation: 142

I was able to keep the old load file in s3 with a bit of a trick at the S3 target end. It is true that AWS DMS doesn't provide anything to keep the old load file after restarting the DMS task. But if you turn on versioning in the target S3 bucket, then you can keep that old load file as a previous version.

This solution was able to fulfill my requirements.

Upvotes: 1

Javier Benitez
Javier Benitez

Reputation: 166

This is said in another thread [here][1]:

For DMS the incremental counter is started over from 1 each time the task is run. It does not have a "Don't override existing objects" feature.

And to this day, its not possible to change the file naming. So, to do this, you are force to have your execution results in different folders. [1]: https://stackoverflow.com/a/60385265

Upvotes: 0

Related Questions