Creating an AWS DMS task using AWS CDK

Question

I am trying to create an AWS DMS task using AWS CDK. But I don't know where to start. I Could not find a good documentation on how I can create the DMS task using CDK. I found the articles on both of the topics but couldn't find an article which adrresses this - talks about how I can create the DMS task using CDK.

Can anyone point me to the right article which explains this or help me do this?

P.S. - I have initialized the project with dms maven dependency. I am using JAVA.

Thanks

Answer 1

You can create an AWS CDK task that captures ongoing changes from the source data store. You can do this capture while you are migrating your data. You can also create a task that captures ongoing changes after you complete your initial (full-load) migration to a supported target data store. This process is called ongoing replication or change data capture (CDC). AWS CDK uses this process when replicating ongoing changes from a source data store. This process works by collecting changes to the database logs using the database engine's native API.

Answer 2

Just addition to previous set up - due to some changes on DMS - it does not wait until the IAM resources are created - so add that as a dependency to subnetgroup resource and add dependency to instance for subnetg and that should save you with 2-3 hours of why it is not working but working in silo's of code....

import * as cdk from '@aws-cdk/core';
import * as dms from '@aws-cdk/aws-dms';
import { 
  ManagedPolicy, 
  Role, 
  ServicePrincipal, 
  PolicyStatement, 
  Effect 
} from '@aws-cdk/aws-iam';

import { App, Construct, Stack } from "@aws-cdk/core";

const app = new App();

app.synth()



export class DmsStack extends cdk.Stack {
  role: Role;
  public constructor(scope:cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    
    const dmsVPCServiceRole = new Role(this, 'dms-vpc-role', {
      assumedBy: new ServicePrincipal('dms.amazonaws.com'),
      roleName: 'dms-vpc-role'
    });

    // Add a policy to a Role
    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          'sts:AssumeRole',
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          'dms:*',
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          "kms:ListAliases", 
          "kms:DescribeKey"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          "iam:GetRole",
          "iam:PassRole",
          "iam:CreateRole",
          "iam:AttachRolePolicy"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [       
          "ec2:CreateVpc", 
          "ec2:CreateSubnet",      
          "ec2:DescribeVpcs",
          "ec2:DescribeInternetGateways",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:ModifyNetworkInterfaceAttribute",
          "ec2:CreateNetworkInterface",
          "ec2:DeleteNetworkInterface"
        ]
      })
    );


    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:FilterLogEvents",
          "logs:GetLogEvents"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['arn:aws:s3:::BUCKETNAME/*'],
        actions: [            
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:PutObjectTagging"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['arn:aws:s3:::BUCKETNAME'],
        actions: [            
          "s3:ListBucket"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['arn:aws:s3:::BUCKETNAME'],
        actions: [            
          "s3:GetBucketLocation"
        ]
      })
    );

    const dmsVpcManagementRolePolicy = ManagedPolicy.fromManagedPolicyArn(
      this, 
      'AmazonDMSVPCManagementRole', 
      'arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole'
    );

    dmsVPCServiceRole.addManagedPolicy(dmsVpcManagementRolePolicy);





    
    // // Create a subnet group that allows DMS to access your data
    const subnet = new dms.CfnReplicationSubnetGroup(this, 'SubnetGroup', {
      replicationSubnetGroupIdentifier: 'cdk-subnetgroup',
      replicationSubnetGroupDescription: 'Subnets that have access to my data source and target.',
      subnetIds: ['subnet-01', 'subnet-02']
    });

    subnet.node.addDependency(dmsVPCServiceRole);



    const instance = new dms.CfnReplicationInstance(this, 'Instance', {
      replicationInstanceIdentifier: 'cdk-instance',

      // Use the appropriate instance class: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Types.html
      replicationInstanceClass: 'dms.t2.small',

      // Setup networking
      replicationSubnetGroupIdentifier: subnet.replicationSubnetGroupIdentifier,
      vpcSecurityGroupIds: [ 'sg-041c1c796c1130121' ],
    });

    instance.node.addDependency(subnet)


  }
}

Answer 3

There are no CDK constructs to simplify working with DMS. Therefore you'll have to use the CloudFormation resources: CfnEndpoint, CfnReplicationTask, etc.

I'm providing the following example to get you started, but be warned, the DMS CloudFormation resources are quite challenging.

import * as cdk from '@aws-cdk/core';
import * as dms from '@aws-cdk/aws-dms';

export class DmsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create a subnet group that allows DMS to access your data
    const subnet = new dms.CfnReplicationSubnetGroup(this, 'SubnetGroup', {
      replicationSubnetGroupIdentifier: 'cdk-subnetgroup',
      replicationSubnetGroupDescription: 'Subnets that have access to my data source and target.',
      subnetIds: [ 'subnet-123', 'subnet-456' ],
    });

    // Launch an instance in the subnet group
    const instance = new dms.CfnReplicationInstance(this, 'Instance', {
      replicationInstanceIdentifier: 'cdk-instance',

      // Use the appropriate instance class: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Types.html
      replicationInstanceClass: 'dms.t2.small',

      // Setup networking
      replicationSubnetGroupIdentifier: subnet.replicationSubnetGroupIdentifier,
      vpcSecurityGroupIds: [ 'sg-123' ],
    });

    // Create endpoints for your data, see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dms-endpoint.html
    const source = new dms.CfnEndpoint(this, 'Source', {
      endpointIdentifier: 'cdk-source',
      endpointType: 'source',
      engineName: 'mysql',

      serverName: 'source.database.com',
      port: 3306,
      databaseName: 'database',
      username: 'dms-user',
      password: 'password-from-secret',
    });

    const target = new dms.CfnEndpoint(this, 'Target', {
      endpointIdentifier: 'cdk-target',
      endpointType: 'target',
      engineName: 's3',

      s3Settings: {
        bucketName: 'target-bucket'
      },
    });

    // Define the replication task
    const task = new dms.CfnReplicationTask(this, 'Task', {
      replicationInstanceArn: instance.ref,

      migrationType: 'full-load', // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dms-replicationtask.html#cfn-dms-replicationtask-migrationtype
      sourceEndpointArn: source.ref,
      targetEndpointArn: target.ref,
      tableMappings: JSON.stringify({
        "rules": [{
          "rule-type": "selection",
          "rule-id": "1",
          "rule-name": "1",
          "object-locator": {
            "schema-name": "%",
            "table-name": "%"
          },
          "rule-action": "include"
        }]
      })
    })
  }
}