How to download all the contents of an S3 bucket and then upload to another bucket?

Question

• I need to get the contents of one S3 bucket into another S3 bucket.
• The buckets are in two different accounts.
• I was told not to create a policy to allow access to the destination bucket from the origin bucket.

Using the AWS CLI how can I download all the contents of the origin bucket, and then upload the contents to the destination bucket?

Answer 1

The aws cli allows you to configure named profiles which lets you use a different set of credentials for each individual cli command. This will be helpful because your buckets are in different accounts.

To create your named profiles you'll need to make sure you already have IAM users in each of your accounts and each user will need a set of access keys. Create your two named profiles like this.

aws configure --profile profile1
aws configure --profile profile2

Each of those commands will ask you for your access keys and a default region to use. Once you have your two profiles, use the aws cli like this.

aws s3 cp s3://origin /local/path --recursive --profile profile1
aws s3 cp /local/path s3://destination --recursive --profile profile2

Notice that you can use the --profile parameter to tell the cli which set of credentials to use for each command.

Answer 2

To copy locally

aws s3 sync s3://origin /local/path

To copy to destination bucket:

aws s3 sync /local/path s3://destination