CODEWITHSUNDEEP
amazon-web-servicesimportexportaws-dms
user2833608
user2833608

Reputation: 51

How AWS DMS works internally

In AWS DMS how does the migration happening internally? Is it like exporting entire data from source table and importing to destination table? Or is it like migrating table records one by one to destination table? I am new to aws dms and don't have much idea on how things work there.

Upvotes: 3

Views: 4007

Answers (4)

TusharK
TusharK

Reputation: 91

When I first used DMS I had same question. So simply I enabled Cloudwatch logs and created one migration task from Oracle to Aurora Postgresql.

  • First DMS task runs on Replication Instance and it connects to Source and Target databases.
  • RI then connect to Source database and based on selection rule it identifies tables and column details since it has lot of special access on Source and Target DB.
  • After that it start reading source table(s) in parallel and create Select col1, col2, col3.. from kind of query to fetch data from Source.
  • Then it write files in a temp location on RI based on tables, 1 file per table and approx 10000 rows in one commit.
  • While all this is happening another process is creating connection to Target DB and checking if Tables already exist if yes then it check which option we selected Do Nothing or Truncate Table etc.. Based on that it takes action.
  • Till now we have data from Source table in files on RI and connection and tables created on Target DB. Now RI just reads file records from RI temp location and create insert query.
  • Once last commit is successful it deletes the temp file from RI.
  • Once Source table and target table count is matched it closes connections in case of One time load.
  • In case of On going changes it keeps connection alive and read redo logs or other logs in Source db. Then follow same process mentioned above for CDC.

Upvotes: 1

dz902
dz902

Reputation: 5838

In addition to above answers, DMS uses Attunity underneath. There are public documents on how the later works in detail.

Upvotes: 0

alan
alan

Reputation: 6953

Here's a doc that provides some more information on how DMS Ongoing Replication works internally: https://aws.amazon.com/blogs/database/introducing-ongoing-replication-from-amazon-rds-for-sql-server-using-aws-database-migration-service/

The short of it is:

(following some initial steps) AWS DMS does not use any replication artifacts. When all the required information is available in the transaction log or transaction log backup, AWS DMS uses the fn_dblog() and fn_dump_dblog() functions to read changes directly from the transaction logs or transaction log backups using the log sequence number (LSN).

Upvotes: 1

JimmyL
JimmyL

Reputation: 418

AWS publish how DMS works in their documentation and blog posts. This is the list I wish I had when I started with DMS:

For a high level understanding see: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.html

A task can consist of three major phases:

  • The full load of existing data
  • The application of cached changes
  • Ongoing replication

During a full load migration, where existing data from the source is moved to the target, AWS DMS loads data from tables on the source data store to tables on the target data store. While the full load is in progress, any changes made to the tables being loaded are cached on the replication server; these are the cached changes.

...

When the full load for a given table is complete, AWS DMS immediately begins to apply the cached changes for that table. When all tables have been loaded, AWS DMS begins to collect changes as transactions for the ongoing replication phase. After AWS DMS applies all cached changes, tables are transactionally consistent. At this point, AWS DMS moves to the ongoing replication phase, applying changes as transactions.

From: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Components.html

Look at the headings:

  1. Replication Tasks
  2. Ongoing replication, or change data capture (CDC)

To gain a detailed understanding of how DMS works internally, read through the following blogs from AWS:

  1. Debugging Your AWS DMS Migrations: What to Do When Things Go Wrong (Part 1)
  2. Debugging Your AWS DMS Migrations: What to Do When Things Go Wrong (Part 2)
  3. Debugging Your AWS DMS Migrations: What to Do When Things Go Wrong? (Part 3)

Finally, work through the blogs particular to your source and target databases at https://aws.amazon.com/blogs/database/category/migration/aws-database-migration-service-migration/

Upvotes: 2

Related Questions