CODEWITHSUNDEEP
amazon-web-servicesmetadatadatabase-replicationcdcaws-dms
Jake Boomgaarden
Jake Boomgaarden

Reputation: 3586

AWS DMS Task Migration Mapping Rules - Add-Column using metadata

I need to create a new column for each record that DMS processes, right now I have the following mapping rule:

{
      "rule-type": "transformation",
      "rule-id": "2",
      "rule-name": "2",
      "rule-target": "column",
      "object-locator": {
        "schema-name": "my_schema",
        "table-name": "%"
      },
      "rule-action": "add-column",
      "value": "_my_column",
      "expression": "'test_val_'||$AR_M_SOURCE_SCHEMA||'.'||$AR_M_SOURCE_TABLE_NAME",
      "data-type": {
        "type": "string",
        "length": 200
      }
    }

and it validates correctly and runs a full load and CDC without error. The problem is that the resulting column value for _my_column looks like test_val_.

I followed the documentation here and it seems to very clearly indicate that these values should not be empty strings.

You can add the metadata information to the target table by using the expressions following:

$AR_M_SOURCE_SCHEMA – The name of the source schema. $AR_M_SOURCE_TABLE_NAME – The name of the source table. $AR_M_SOURCE_COLUMN_NAME – The name of a column in the source table. $AR_M_SOURCE_COLUMN_DATATYPE – The data type of a column in the source table.

I can't find any other resource from AWS that indicates how I would get the source schema and table values for the mapping. I can see when the data is inserted into the target database that there is metadata and it does include the schema & table names, so the information exists at the time the record is written.

I know that I can include the schema in the literal string since I am already setting it somewhere else in the mapping, but I would need to add this to every table in the schema, so I need that value from the metadata while it is processing.

I must be missing something obvious, does anyone know what the issue here is?

Upvotes: 0

Views: 973

Answers (1)

Jake Boomgaarden
Jake Boomgaarden

Reputation: 3586

found the issue. I don't know why and there is nothing in the documentation that indicates this (that I could find) but referencing two metadata variables seems to break it and have all metadata variables return an empty string value.

      "expression": "'test_val_myschema'.'||$AR_M_SOURCE_TABLE_NAME",

works. Hopefully this answer can help someone else who gets stuck here

Upvotes: 1

Related Questions