CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3aws-lambdaboto3
Tesuji
Tesuji

Reputation: 1675

Why am I getting different errors when trying to read s3 key that does not exist

I have two environments. One, I am running from desktop, and one I am running from an aws lambda. Both of these are python. The code I have on my machine is:

import boto3
s3 = boto3.resource('s3')
bucket = 'my-bucket'
obj = s3.Object(bucket,'test.txt')
try:
    text = obj.get()['Body'].read().decode()
    print("file exists"))
except s3.meta.client.exceptions.NoSuchKey:
    print("key does not exist")

When I run this, everything works perfectly fine, as intended.

When I run this on the lambda, I get

An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

My lambda permissions are correct, so it can't be that, as everything works when the file exists. I am deleting the file each run so it has nothing to do with the order I run my test code. To debug, I did this:

except Exception as e:
    print(e)

In both my lambda code and local code, to see what the actual error is. In my local machine, the error is:

An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

When I run it on my lambda, it is

An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

Why am I getting different error messages, and how do I account for this without a lazy "except all Exceptions" solution? Thank you

This is the CFT for my lambda

                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:PutObject",
                    "s3:GetObject",
                    "s3:ListBucket",
                  ],
                  "Resource": {
                    "Fn::Sub": "arn:aws:s3:::my-bucket/*"
                  }

                }

Upvotes: 9

Views: 7554

Answers (1)

rdas
rdas

Reputation: 21275

From: https://forums.aws.amazon.com/thread.jspa?threadID=56531

As you discovered, Amazon S3 will return an AccessDenied error when a nonexistent key is requested and the requester is not allowed to list the contents of the bucket. By the Amazon S3 definition, not being allowed to list the contents of a bucket means not being allowed to discover whether a particular key exists. Returning NoSuchKey would leak information about the nonexistence of the requested key. Instead, Amazon S3 returns AccessDenied. AccessDenied does not say anything about the existence or nonexistence of the requested key, so no information is leaked.

You're probably missing the permission to list the contents of the bucket (s3:listBucket) in lambda.

Ref: https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-buckets

As per comment, make sure you write the resource name correctly:

the resource for s3:ListBucket has to be the bucket itself "arn:aws:s3:::BucketName", unlike the resource for GetObject and PutObject which is (in my case) all objects of the bucket "arn:aws:s3:::BucketName/*"

Upvotes: 21

Related Questions