CODEWITHSUNDEEP
amazon-s3
David McLaughlin
David McLaughlin

Reputation:

S3: Duplicate buckets

What is the easiest way to duplicate an entire Amazon S3 bucket to a bucket in a different account?

Ideally, we'd like to duplicate the bucket nightly to a different account in Amazon's European data center for backup purposes.

Upvotes: 7

Views: 3178

Answers (6)

Sean
Sean

Reputation: 152

Note: this doesn't work for cross-account syncing, but this works for cross-region on the same account.

For simply copying everything from one bucket to another, you can use the AWS CLI (https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/): aws s3 sync s3://SOURCE_BUCKET_NAME s3://NEW_BUCKET_NAME

In your case, you'll need the --source-region flag: https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

If you are moving an enormous amount of data, you can optimize how quickly it happens by finding ways to split the transfers into different groups: https://aws.amazon.com/premiumsupport/knowledge-center/s3-large-transfer-between-buckets/

There are a variety of ways to run this nightly. One is example is the AWS instance-schedule (personally unverified) https://docs.aws.amazon.com/solutions/latest/instance-scheduler/appendix-a.html

Upvotes: 1

Yahya Younes
Yahya Younes

Reputation: 710

You can make an application or service that responsible to create two instances of AmazonS3Client one for the source and the other for the destination, then the source AmazonS3Client start looping in the source bucket and streaming objects in, and the destination AmazonS3Client streaming them out to the destination bucket.

Upvotes: 1

JerSchneid
JerSchneid

Reputation: 5917

I suspect there is no "automatic" way to do this. You'll just have to write a simple app that moves the files over. Depending on how you track the files in S3 you could move just the "changes" as well.

On a related note, I'm pretty sure Amazon does a darn good job backup up the data so I don't think you necessarily need to worry about data loss, unless your back up for archival purposes, or you want to safeguard against accidentally deleting files.

Upvotes: 1

Chap
Chap

Reputation: 3553

If you're worried about deletion, you should probably look at S3's new Versioning feature.

Upvotes: 1

David McLaughlin
David McLaughlin

Reputation:

Cool, I may look into writing a script to host on Ec2. The main purpose of the backup is to guard against human error on our side -- if a user accidentally deletes a bucket or something like that.

Upvotes: 1

Michael Burr
Michael Burr

Reputation: 340208

One thing to consider is that you might want to have whatever is doing this running in an Amazon EC2 VM. If you have your backup running outside of Amazon's cloud then you pay for the data transfer both ways. If you run in an EC2 VM, you pay no bandwidth fees (although I'm not sure if this is true when going between the North American and European stores) - only for the wall time that the EC2 instance is running (and whatever it costs to store the EC2 VM, which should be minimal I think).

Upvotes: 3

Related Questions