Migrate from Digital Ocean Space to Amazon AWS

Question

The goal

I need to move thousands of objects from Digital Ocean Space to Amazon AWS. All files are images and the total size is around 36GBs.

The problem i'm facing:

Googling around I can see we can transfer from Amazon AWS to Digital Ocean Spaces, but not the other way around.

What I'v found

All codes I find are about upload several objects to a bucket but not download them from a bucket.

Answer 1

I had trouble using rclone. It was too complex so I built a PHP script to do this automatically. You need to have API key/secret for both DO Space/AWS Bucket and that's it.

https://gitlab.com/exevio-open-source/digitalocean-spaces-transfer

Worked great for my company when we migrated accounts with over 2TB of files.

Answer 2

I already accepted John Rotestein's answer, but Digital Ocean answered my question as well.

There's a way to download all Digital Ocean spaces content in Windows with Cyberduck:

https://www.digitalocean.com/community/tutorials/how-to-manage-digitalocean-spaces-with-cyberduck

Than you can build a script to loop your directory and send objects to Amazon AWS. If you don't know how, you can start here and here.

Update Cyberduck stucks after a while if you have too many files. SO i'm using S3EXPLORER from this link which also has installation instructions :- http://s3browser.com/download.aspx

Answer 3

Based upon Digital Ocean's article How To Migrate from Amazon S3 to DigitalOcean Spaces with rclone, it seems that you can use rclone to copy between the two services (in either direction).