AWS CLI S3 sync bucket and keep permissions

Question

I need to do a mirror copy of bucket within my amazon account.

The main problem is that some of objects have "private" permissions, while other part have "public-read" permissions.

When I run:

aws s3 sync s3://bucket-saas s3://bucket-saas-bkp --acl public-read

All objects become public, while when I run

aws s3 sync s3://bucket-saas s3://bucket-saas-bkp --acl private All objects become private.

Is there any way to sync files and keep their permissions?

Answer 1

I've encountered the same issue - I had a bucket with tons of objects, while some of them should be publicly accessible. I had to copy entire bucket to another one while preserving the ACLs and of course, setting up the ACLs manually would take me a hell of time.

I've made this simple script in python which copies the objects from one bucket to another and also sets up the ACLs for it.

Feel free to take a look: https://github.com/terminator9999/aws-s3-bucket-copy/

Answer 2

--acl is a canned ACL, which is a predefined grant. You can see what canned ACLs are available in the AWS Documentation for Canned ACLs. Sadly it doesn't look like there is an option to keep existing permissions when you copy objects between buckets via the CLI. Based off the bucket names, I'm assuming you're trying to backup one bucket to another. Amazon does offer a feature called Cross Region Replication which is what I think you're looking for. CRR is perfect for this as it will copy across each object just about instantly to the backup bucket while preserving a lot of the data associated with that object.

From the CRR documentation:

The object replicas in the destination bucket are exact replicas of the objects in the source bucket. They have the same key names and the same metadata—for example, creation time, owner, user-defined metadata, version ID, access control list (ACL), and storage class. Optionally, you can explicitly specify a different storage class for object replicas. And regardless of who owns the source bucket or the source object, you can choose to change replica ownership to the AWS account that owns the destination bucket.

If that's not what you're trying to do and you just want a script to sync the objects between buckets and retain permissions, you're probably going to have to write a script that will do a lookup of each object after its copied it over to the new bucket and assign the existing permissions to it. Keep in mind you do have to pay for these S3 API operations so I would advise you to do your research so you don't get any nasty surprises on your AWS bill.