CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
Nick
Nick

Reputation: 189

Can I replicate data from Multiple S3 buckets to a single S3 bucket?

I ave a requirement where I want to copy/merge data from multiple S3 buckets to a single S3 bucket natively. I can do this using Glue or other AWS services. However, I want to see if this can be done natively using S3 cross region replication

Bucket 1 ---> Bucket A
Bucket 2 ---> Bucket A
Bucket 3 ---> Bucket A

The buckets could be in the same or different regions.

Thanks

Upvotes: 0

Views: 1357

Answers (1)

b.b3rn4rd
b.b3rn4rd

Reputation: 8830

Recently I was trying to achieve the same outcome via S3 Replication, in my particular example I was trying to overwrite the same object in the hub bucket from multiple spoke buckets, unfortunately that setup is not support (confirmed with AWS SA)

This is the setup diagram

Bucket B -> Bucket A (edl.txt)
Bucket C -> Bucket A (edl.txt)

Once the object edl.txt is replicated into Bucket A from Bucket B, it's state is REPLICA is it can't be further updated via replication from the Bucket C

The workaround solution would be to create a lambda function triggered via S3 event in the account where Bucket A is located that would re-upload eld.txt object back to Bucket A so it's not consider as a REPLICA anymore but appears as a locally uploaded file. In this case it can be further updated via replication

Upvotes: 1

Related Questions