CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-cli
Tasawar Hussain
Tasawar Hussain

Reputation: 765

aws s3 mv/sync command

I have about 2 million files nested in subfoldrs in a bucket and want to move all of them to another bucket. Spending much of time on searching ... i found a solution to use AWS CLI mv/sync command. use move command or use sync command and then delete all the files after successfully synced.

aws s3 mv s3://mybucket/ s3://mybucket2/ --recursive

or it can be as

aws s3 sync s3://mybucket/ s3://mybucket2/

But the problem is how would i know that how many files/folders have moved or synced and how much time would it take... And what if some exception occurs(machine/server stops/ internet disconnection due to any reason )...i have to again execute the command or it will for surely complete and move/sync all files. How can i be sure about the number of files moved/synced and files not moved/synced.

or can i have something like that

I move limited number of files e.g 100 thousand.. and repeat until all files are moved...

or move files on the basis of uploaded time.. e.g files uploaded from starting date to ending date

if yes .. how?

Upvotes: 2

Views: 2970

Answers (1)

Peycho Dimitrov
Peycho Dimitrov

Reputation: 1407

To sync them use:

aws s3 sync s3://mybucket/ s3://mybucket2/

You can repeat the command, after it finish (or fail) without issue. This will check if anything is missing/different to the target s3 bucket and will process it again.

The time depends on what size are the files, how much objects you have. Amazon counts directories as an object, so they matter too.

Upvotes: 4

Related Questions