CODEWITHSUNDEEP
postgresqlamazon-s3parquetcdcaws-dms
SaadK
SaadK

Reputation: 256

AWS DMS CDC task does not detect column name and type changes

I have created a CDC task that captures changes in a source PostgreSQL schema and writes them in Parquet format into a target S3 bucket. The task captures the inserts, updates and deletes correctly but fails to capture column name and type changes in the source.

When I change a column name or type of a table in the source and insert new rows to the table, the resulting Parquet file uses the old column name and type.

Is there a specific configuration I am missing? or it is not possible to achieve the desired outcome from this task in DMS?

Upvotes: 3

Views: 3501

Answers (2)

Anner
Anner

Reputation: 21

Make sure you have the HandleSourceTableAltered parameter set to true in the task settings.[1] (The setting applies when the target metadata parameter BatchApplyEnabled is set to either true or false.)

Same goes for HandleSourceTableDropped or HandleSourceTableTruncated if this is relevant in your case.

Obviously, previously replicated Parquet files on S3 will not change to reflect this DDL change on the source.

[1] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.DDLHandling.html

Upvotes: 1

GRVPrasad
GRVPrasad

Reputation: 1142

if you change column at source and DMS will pick automatically from source and update at destination. check your DMS setting. you no need to do manually adding column at destination

enter image description here

Upvotes: 0

Related Questions