CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
Dick McManus
Dick McManus

Reputation: 789

Copy S3 files to another S3 in a different account as they land

I want to set up a bucket for testing, where I pull in a previous month of data from an S3 bucket in a different AWS account, and then continue to consume data from that S3 bucket as it lands.

All the documentation I am seeing talks about full copies or syncs between buckets, and there is far too much data in the initial bucket for that. I need to be able to just pull fresh data in as it lands. I'm not sure what the best method for this would be, as it would be likely a several thousand files a day.

Upvotes: 0

Views: 55

Answers (1)

coder
coder

Reputation: 4429

You should look at S3 replication for your use case to copy data from primary bucket to secondary bucket. S3 replication will copy only the new data from the primary bucket to secondary bucket.

However the secondary bucket will have to be in a different AWS region. (Hopefully thats not the limiting case for you)

check this link for setup (https://medium.com/@chrisjerry9618/s3-cross-region-replication-2e20f2dc86e0)

Upvote this or mark as answer for others to get help

Upvotes: 2

Related Questions