S3 Website Cross Account Permissions Not Working

Question

Quick Summary:

If objects are put into a bucket owned by "Account A" from a different account ("Account B"), you cannot access files via S3 static website (http) from "Account A" (bucket owner).

This is true regardless of the bucket policy granting GetObject on all objects, and regardless of if bucket-owner-full-control ACL is enabled on the object.

• If trying to download a file from Account A via S3 API (console/cli), it works fine.
• If trying to download a file from Account A via S3 static website (http), S3 responds HTTP 403 Forbidden if the file was uploaded by Account B. Files uploaded by Account A download fine.
• Disabling Object ACL's fixes the problem but is not feasible (explained below)

PROBLEM OVERVIEW

I have a unique setup where I need to publish files to an S3 bucket from an account that does not own the bucket.

The upload actions work fine. My problem is that I cannot access files from the bucket-owner account over the S3 static website if the files were published from another account (403 Forbidden response).

The problem only exists if the files were pushed to S3 FROM a different account. Because the issue is only for those files, the problem seems like it would be in the Object Ownership ACL configuration. I've confirmed I can access other files (that weren't uploaded by the other acct) in the bucket through the S3 static website endpoint, so I know my bucket policy and VPC endpoint config is correct.

If I completely disable Object ACL's completely it works fine, however I cannot do that because of two issues:

• Ansible does not support publishing files to buckets with ACL's disabled. (Disabling ACL is a relatively new S3 feature and Ansible doesn't support it)
• The primary utility I'm using to publish files (Aptly) also doesn't support publishing to buckets with ACL's disabled. (Disabling ACL is a relatively new S3 feature and Aptly doesn't support it)

Because of these above constraints, I must use Object ACL's enabled on the bucket. I've tried both settings "Object Writer" and "Bucket owner preferred", neither are working. All files are uploaded with the bucket-owner-full-control object ACL.

As mentioned, disabling ACL fixes everything, but since my client tools (Ansible and Aptly) cannot upload to S3 without an ACL set, ACL's must remain enabled. (Disabling ACL is a relatively new S3 feature, added late 2021)

ENVIRONMENT EXPLAINED:

• Bucket test-bucket-a is in "Account A", it's not a "private" bucket but it does not allow public access. Access is granted via policies (snippet below).

• Bucket objects (files) are pushed to test-bucket-a from an "Account B" role.

  • Access from "Account B" to put files into the bucket is granted via policies (not shown here). Files upload without issue.
  • Objects are given the bucket-owner-full-control ACL when uploading.

• I have verified that the ACL's look correct and both "Account A" and "Account B" have object access. (screenshot at bottom of question)

• I am trying to access the files from the bucket-owner account (Account A) over the S3 static website access (over http). I can access files that were not uploaded by "Account B" but files uploaded by "Account B" return 403 Forbidden

Here is an example of how this file is uploaded using Ansible:

Reminder: the role doing the uploading is NOT part of the bucket owner account.

- name: "publish gpg pubkey to s3 from Account B"
  aws_s3:
    bucket: "test-bucket-a"
    object: "/files/pubkey.gpg"
    src: "/home/file/pubkey.gpg"
    mode: "put"
    permission: "bucket-owner-full-control"

I am using VPC Endpoint to access, and this is added to the bucket policy. All the needed routes and endpoint config are in-place. I know my policy config is good because if I disable the Object ACL everything works perfectly.

{
    "Sid": "AllowGetThroughVPCEndpoint",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::test-bucket-a/*",
    "Condition": {
      "StringEquals": {
        "aws:sourceVpce": "vpce-0bfb94<scrubbed>"
      }
    }
},

Some key troubleshooting notes:

• From "Account A" when logged into the console, I can download the file. This is very strange and shows that API requests to GetObject are working. Does the S3 website config follow some different rule structure??
• From "Account A" when accessing the file from an HTTP endpoint (S3 website) it returns HTTP 403 Forbidden
• I have tried deleting and re-uploading the file multiple times.
• I have tried manually setting object ACL via the aws cli (ex: aws s3api put-object-acl --acl bucket-owner-full-control ...)
• Giving "public" ACL's such as public-read and public-read-write solves the access problem, but the bucket contents should only be accessible from within the VPC
• When viewing the "object" ACL, I have confirmed that both "Account A" and "Account B" have access. See below screenshot. Note that it confirms the object owner is an external account.

Answer 1

This problem was Ansible's fault the whole time.

Even though permission: "bucket-owner-full-control" was set during upload, the setting wasn't working properly. I believe as a side effect of this issue: https://github.com/ansible-collections/amazon.aws/issues/219

Fix:

I ended up doing an ansible-galaxy collection install --upgrade amazon.aws and tried re-uploading the file and everything worked as expected.