CODEWITHSUNDEEP
mongodbamazon-web-servicesamazon-s3amazon-dynamodbaws-dms
4d61726b
4d61726b

Reputation: 497

AWS mirgate data from MongoDB to DynamoDB/S3/Redshift

The issue is that mirgating data from MongoDB to DynamoDB/S3/Redshift currently, as I unterstand for us is not available via AWS DMS Service, as it does not support all data types. Or maybe I'm wrong.

The probelm is that our Mongo object contain not scalar fields(arrays, maps). So when I make a mirgation task via AWS DMS with table mode, it pull data badly.Buy some reason only selection works. Transformation rules are ignored by DMS(tried renaming and removing).

In the doc mode is all ok, but how can I run migration with some custom script for transformation? As storing data this way still need transformation.

We need some modifications like: rename, remove fields and flatting some fields(for example we ahve a map object and it should be flatten into several scalar fields).

Migration should be done into one of the sources: S3, Dyanamo, Redshift

Will be thankfull for any help and suggestions.

Upvotes: 2

Views: 625

Answers (1)

BigData-Guru
BigData-Guru

Reputation: 1261

use the following below script to take a backup of the MongoDB DB

mongodump -h localhost:27017 -d my_db_name -o $DEST

use the below command to sync your backup to S3 bucket

aws s3 sync ~/db_backups s3://my-bucket-name

Once your data in S3, you can load very easily to Redshift using copy command

Upvotes: 2

Related Questions