CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
Filip Stefanov
Filip Stefanov

Reputation: 842

Cannot download objects from S3 after adding full-control-header

Scenario:

I have bucket with public read/write based on IP condition.

When anonymous user uploads a file the owner is 65a011a29cdf8ec533ec3d1ccaae921c I cannot rename/move/copy such files because I'm lacking permissions and bucket replication is not working for them for the same reason. However i can delete (because I'm bucket owner) and everyone else from whitelisted IP's can download them using the link S3 provides.

This header was added to the script used to upload stuff to address the issue with replication.

{'x-amz-grant-full-control': 'id=my canonical id'}

Now i have full control over the objects and replication is working fine but i cannot download them using the link from S3 or wget or CLI in fact no one can...

HTTP request sent, awaiting response... 403 Forbidden
2017-09-22 00:04:32 ERROR 403: Forbidden.

Only download button from file overview in S3 web interface seems to work.

Why after applying this ACL, objects does not comply with bucket policy anymore?

Upvotes: 0

Views: 2412

Answers (2)

Filip Stefanov
Filip Stefanov

Reputation: 842

Hello Jamie thanks for the concise answer, i came up with this script to change the owner from anonymous to bucket owner:

#!/usr/bin/env bash

ARRAY=($(aws s3 ls --recursive s3://mybucket/folder/ | awk '{print $4}'))

for key in "${ARRAY[@]}"; do
  echo "$key";
  aws s3api put-object-acl \
  --bucket mybucket        \
  --key "$key"             \
  --grant-full-control     \
  --no-sign-request        \
  "id=bucket owner canonical id"; done

aws s3 mv s3://mybucket/folder/ s3://mybucket/folder-copy/ --recursive
sleep 10;
aws s3 mv s3://mybucket/folder-copy/ s3://mybucket/folder/ --recursive

It took 68m45.076s for ~150GB of data. Not planning to use anonymous uploads anymore :)

Upvotes: 1

Jamie Starke
Jamie Starke

Reputation: 9234

Alright, I thought this might be the case, but wanted to confirm some details.

S3 has two different permission models: Access Control Lists (ACL) and Bucket Policies.

Access Control Lists

Every object in S3 has it's own Access Control List (ACL). With this, individual users can granted read access to an individual file. This can also grant access to the Bucket Owner to be able to control the files.

Bucket Policies

This uses a more descriptive permission format, similar to that of IAM Policies. With it, access can be granted for cross account access, to specific IAM Users and Roles, etc.

Bucket Policies are only observed when the owner of the object is also the owner of the bucket.

Options

  • Have your script also grant ACL Public read on the objects
  • Use signed urls generated by your application (EC2, Lambda) to allow users to securely upload on your behalf
  • Have object owner dynamically changed. This would involve a lambda function that executes on PutObject, and if the bucket owner and object owner are different, copies the object back.

Upvotes: 1

Related Questions