CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambda
AlonAlmog
AlonAlmog

Reputation: 1

Syncing files over different accounts buckets

I’m trying to sync one aws bucket to an another bucket across different iam accounts. How can I do it periodically so any file written to the source bucket will automatically transforms to the destination? Do I need to use lambdas to execute aws cli sync command?

Thanks

Upvotes: 0

Views: 167

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269826

Option 1: AWS CLI Sync

You could run aws s3 sync on a regular basis, which will only copy new/changed files. This makes it very efficient. However, if there is a large number of files (10,000+) then it will take a long time trying to determine which files need to be copied. You will also need to schedule the command to run somewhere (eg a cron job).

Option 2: AWS Lambda function

You could create an AWS Lambda function that is triggered by Amazon S3 whenever a new object is created. The Lambda function will be passed details of the Bucket & Object via the event parameter. The Lambda function could then call CopyObject() to copy the object immediately. The advantage of this method is that the objects are copied as soon as they are created.

(Do not use an AWS Lambda function to call the AWS CLI. The above function would be called for each file individually.)

Option 3: Amazon S3 Replication

You can configure Amazon S3 Replication to automatically replicate newly-created objects between the buckets (including buckets between different AWS Accounts). This is the simplest option since it does not require any coding.

Permissions

When copying S3 objects between accounts, you will need to use a single set of credentials that has both Read permission on the source bucket and Write permission on the target bucket. This can be done in two ways:

  • Use credentials (IAM User or IAM Role) from the source account that have permission to read the source bucket. Create a bucket policy on the target bucket that permits those credentials to PutObject into the bucket. When copying, specify ACL=public-read to grant object ownership to the destination account.

OR

  • Use credentials from the target account that have permission to write to the target bucket. Create a bucket policy on the source bucket that permits those credentials to GetObject from the bucket.

Upvotes: 0

Related Questions