Why does AWS DMS occasionally encounter various kinds of errors?

Question

The important things about this question are:

• Occasionally. It works for some time, and encounters errors for other times.
• Errors are all over the place
  • S3 connection failed error (unable to connect to S3 endpoint, failed to list S3 bucket), which really should not happen because it is mostly internal
  • Source database connection error, where source DB is on-premise and connected to AWS via DX
  • There are errors but console shows replication ongoing, missing a large amount of data
  • Source DB says connection is busy, but the load was quite light
  • Source DB says there are LSN related errors
  • Some tables just won't work but others are fine, those tables are ordinary ones, nothing special
  • RAM usage fluctuates a lot, CPU stays high, disk usage stays very high even long after full loading

Remember it DOES work normally for some time, then fail without a valid reason.

Answer 1

I have Googled heavily for this very strange case and was involved with AWS experts and support, but was to no avail.

Then I saw the t2 instance that was used as replication instance and thought maybe that was the source of the problems.

...AND IT WAS.

When we changed t2 to r5, all the problems were gone.

It seems that after the credits had burnt out, DMS could not handle the sudden throttling of the CPU power and that led to various and occasional problems.

Nothing else was changed. Just all the tasks are now moved to the r5 instance and running fine.

So I know t is for "trouble". STAY AWAY FROM T2/T3 unless you really know what you may get into.