CODEWITHSUNDEEP
amazon-web-servicescsvamazon-s3amazon-rdsdms
Jabbar Ahmed
Jabbar Ahmed

Reputation: 1

AWS DMS - How to write to RDS table data to a single S3 target file?

I have successfully setup DMS to copy data from RDS (SQL Server) to S3 in csv format (Full load). However, upon running the task, DMS copies the source table and creates multiple csv files in S3 for the single table. Is there any way to make sure that for 1 table, DMS only creates one target csv file in S3?

Upvotes: 0

Views: 692

Answers (1)

Bơ Loong A Nhứi
Bơ Loong A Nhứi

Reputation: 563

The first full load operation will load all data into one file. For on-going replicated data, migrated data has different format, it contains additional character like this:

  • I: for inserted record
  • U: for changed one
  • D: for deleted one

So, they can not be merged into one file. You can do this by using Lambda, but it's not a good way:

  • Add trigger to Lambda function on S3 bucket whenever any data change is made on above S3 bucket - which contains csv files
  • In Lambda function: handle file in each above cases and merge them in by your self.

I suggest to use other DB target like MySQL, Postgres, etc. As they support them all.

Upvotes: 1

Related Questions