CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3aws-lambda
anonggd
anonggd

Reputation: 41

How to continuously copy new S3 files to another S3 bucket

How can I continuously copy one S3 bucket to another? I want to copy the files every time a new file has been added.

I've tried using the boto3 copy_object however I require the key each time which won't work if I'm getting a new file each time.

Upvotes: 0

Views: 1093

Answers (3)

Oswaldo Zapata
Oswaldo Zapata

Reputation: 693

This is for: How to copy lot of files from one s3 bucket in account A, to another in account B. I know this could be different use case, but for anyone who wants some help on this, here It is.

For small amount of files to copy yeah It could work using aws cli, aws s3 sync, but for buckets with more than 3TB of size and 12 Millions of files It just does not work well. you will deal with other issues like token expiration after 1hour, 12h after configuring the expiration time as the max value allowed.

So, for you use case, I saw by using aws DataSync you can schedule when to run the sync task, I do not think It can be right after adding a new file but It can be after, just configure the task schedule

Schedule options

Here is the documentation, just follow the steps: https://docs.aws.amazon.com/datasync/latest/userguide/tutorial_s3-s3-cross-account-transfer.html

Upvotes: 0

John Rotenstein
John Rotenstein

Reputation: 270224

From Replicating objects - Amazon Simple Storage Service:

To automatically replicate new objects as they are written to the bucket use live replication, such as Same-Region Replication (SRR) or Cross-Region Replication (CRR).

S3 Replication will automatically create new objects in another bucket as soon as they are created. (Well, it can take a few seconds.)

Alternatively, you could configure the S3 bucket to trigger an AWS Lambda function that uses the CopyObject() command to copy the object to another location. This method is useful if you want to selectively copy files, by having the Lambda function perform some logic before performing the copy (such as checking the file type).

Upvotes: 1

brushtakopo
brushtakopo

Reputation: 1408

Please look at this: https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/

You can use the aws cli s3 sync command to achieve this.

Upvotes: 0

Related Questions