Access Denied Error uploading object to s3 bucket

Question

I'm trying to set up a service that receives a file (image, video) and uploads it to my s3 bucket. We want to move from a public bucket to one that is only accessible by specific users, which my account theoretically has:

Bucket Access Control List

I've updated my code to use my new s3 credentials and bucket (I've used this code to upload to the other s3 based bucket system). But now that I'm pointed at this new bucket all I'm getting is "Access Denied" responses. The inner exceptions are also "Access Denied" until I hit null.

I've tried a number of things so far including adding this policy to the bucket:

{
    "Id": "Policy1553720963443",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1553720961364",
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::<BUCKETNAME>/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<iamid>:user/<me>@<company>"
                ]
            }
        }
    ]
}

This CORS config:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
    <AllowedHeader>Authorization</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

And even adding myself to a group with this policy:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
            "s3:*"
        ],
        "Resource": "*"
    },
    {
        "Sid": "VisualEditor1",
        "Effect": "Allow",
        "Action": "s3:*",
        "Resource": [
            "arn:aws:s3:::<BUCKETNAME>/*",
            "arn:aws:s3:::<BUCKETNAME>"
        ]
    }
]

}

None of them have had any effect on the "Access Denied error"

Can anyone help me out here? Is there any way to get more information about the Access Denied error? I've tried to use fiddler to get the detailed response back from aws, but using fiddler seems to cause a gateway timeout error instead, so that doesn't help.

Any advice would be greatly appreciated. Thanks!

~Josh

Edit, Here's the code I'm trying to upload with:

        private static readonly string serviceURL = ConfigurationManager.AppSettings["ServiceURL"];
    private static readonly string BucketName = ConfigurationManager.AppSettings["BucketName"];
    private static readonly long UploadPartSize = Convert.ToInt64(ConfigurationManager.AppSettings["UploadPartSize"]);

    private static AmazonS3Config config = new AmazonS3Config { ServiceURL = serviceURL, ForcePathStyle = true, RegionEndpoint = RegionEndpoint.USWest2 };
    private readonly AmazonS3Client _s3Client = new AmazonS3Client(config);

private async Task<string> UploadFileAsync(string contentType, Stream stream, long length, UploadVideoParams @params, CancellationToken token)
    {
        using (var fileTransferUtility = new TransferUtility(_s3Client))
        {
            var now = DateTime.UtcNow;
            var key = MakeS3Key(@params);

            try
            {
                await fileTransferUtility.UploadAsync(new TransferUtilityUploadRequest
                {
                    BucketName = BucketName,
                    AutoCloseStream = true,
                    AutoResetStreamPosition = false,
                    InputStream = new WrapperStream(stream, length),
                    PartSize = UploadPartSize,
                    StorageClass = S3StorageClass.Standard,
                    Key = key,
                    CannedACL = S3CannedACL.PublicRead,
                    ContentType = contentType
                }, token).ConfigureAwait(false);

                LogInfo(@params.TransactionId, "Made Upload Async call. Key: " + key);
                return key;
            }
            catch (Exception e)
            {
                await fileTransferUtility.AbortMultipartUploadsAsync(BucketName, now, token).ConfigureAwait(false);
                HandleError(@params.TransactionId, e);
                throw e;
            }
        }
    }

Answer 1

I figured it out! Turns out it was the "CannedACL = S3CannedACL.PublicRead," line on my upload. The bucket was correctly configured to deny any public access, but this line wanted to make the uploaded file publicly accessible.

That line is now:

CannedACL = S3CannedACL.BucketOwnerFullControl,

Because that is the access setting we have for the bucket. I wish the error was a little more specific, but that's what we got. I hope this helps anyone else with this issue!

Answer 2

Can you try something for me and see if it works?
Remove the bucket policy completely off the bucket, go to IAM, find your user/group/role and attach a policy against your user that grants you access to S3. For testing, there should be a managed policy named S3FullAccess, attach that and test. To find out which user/group/role you need to attach this to, run aws sts get-caller-identity in your local cli and it should tell you. Note: If this works and you plan on running this code on an ec2 instance, these permissions need to be assigned to the ec2 instance role.

Now, if that works, remove the full access policy, create a scoped down policy and attach that instead. Best to use the principle of least privilege, but I think for the sake of simplicity to get this to work it's ok to grant full access while we test it works.

The rule of thumb I generally use is if the bucket needs cross-account access, use a bucket policy, if the bucket lives in the same account, use IAM permissions to manage access. This will make management of everthing consistent and easy going forward.