Why does my lambda function get Access Denied trying to access an S3 bucket?

Question

1. Lambda execution role has s3 access to 51 functions including ListBuckets and all other read operations.
2. My S3 bucket has a policy to allow access from the lambda role. (and it's in the same account anyway so i dont think this is required).
3. I even made the bucket public access just for fun.

Here's the lambda code. I'm not sure why bucket.objects.all() fails to access s3.

import json import boto3 s3 = boto3.resource('s3')

def lambda_handler(event, context): print(event)

message = 'Hello {}!'.format(event['Records'][0]['s3']['bucket']['name'])  
print(message)
bucket = s3.Bucket('my-bucketname-demo')
# this works
print(bucket.creation_date)

# this fails on access denied
# Iterates through all the objects, doing the pagination for you. Each obj
# is an ObjectSummary, so it doesn't contain the body. You'll need to call
# get to get the whole body.
for obj in bucket.objects.all():
    key = obj.key
    body = obj.get()['Body'].read()
    print(key)
    print(body)
    
return {
    'statusCode': 200,
    'body': json.dumps('Hello from Lambda!')
}

Here's the error: [ERROR] ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

Lambda Execution Role Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketTagging",
                "s3:GetInventoryConfiguration",
                "s3:GetObjectVersionTagging",
                "s3:ListBucketVersions",
                "s3:GetBucketLogging",
                "s3:ListBucket",
                "s3:GetAccelerateConfiguration",
                "s3:GetBucketPolicy",
                "s3:GetStorageLensConfigurationTagging",
                "s3:GetObjectVersionTorrent",
                "s3:GetObjectAcl",
                "s3:GetEncryptionConfiguration",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetIntelligentTieringConfiguration",
                "s3:GetBucketRequestPayment",
                "s3:GetAccessPointPolicyStatus",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectTagging",
                "s3:GetMetricsConfiguration",
                "s3:GetBucketOwnershipControls",
                "s3:GetBucketPublicAccessBlock",
                "s3:GetBucketPolicyStatus",
                "s3:ListBucketMultipartUploads",
                "s3:GetObjectRetention",
                "s3:GetBucketWebsite",
                "s3:GetJobTagging",
                "s3:GetBucketVersioning",
                "s3:GetBucketAcl",
                "s3:GetObjectLegalHold",
                "s3:GetBucketNotification",
                "s3:GetReplicationConfiguration",
                "s3:ListMultipartUploadParts",
                "s3:GetObject",
                "s3:GetStorageLensConfiguration",
                "s3:GetObjectTorrent",
                "s3:DescribeJob",
                "s3:GetBucketCORS",
                "s3:GetAnalyticsConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetBucketLocation",
                "s3:GetAccessPointPolicy",
                "s3:GetObjectVersion",
                "s3:GetStorageLensDashboard"
            ],
            "Resource": "arn:aws:s3:::my-bucket-demo/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListStorageLensConfigurations",
                "s3:GetAccessPoint",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListAllMyBuckets",
                "s3:ListAccessPoints",
                "s3:ListJobs",
                "s3:ListObjects"
            ],
            "Resource": "*"
        }
    ]
}

Answer 1

I needed to GET to the object without the AWS SDK and I was facing the same issue.

I resolved it by creating a lambda function with a static IP and allow that IP address to GetObject on the S3 bucket.

1. Create a new VPC to run your code - or use an existing VPC - in case you already have a VPC with Private/Public subnet and a NAT Gateway with Elastic IP address, you can go to step 6.
2. Create a new Internet Gateway to communicate with the Internet from inside your VPC.
3. Create a Public Subnet and add a new route to the routing table which routes to your Internet Gateway from 0.0.0.0/0.
4. Create a new Elastic IP address for the static IP. Create a new NAT Gateway and assign it to the Public Subnet and Elastic IP address that you just used.
5. Create a Private Subnet and add a new route to the routing table which routes to your NAT Gateway from 0.0.0.0/0.
6. Create a security group and set your inbound and outbound rules based on your requirements (ideally no inbound traffic and allow all outbound traffic).
7. Attach the VPC, Subnet, security group to the lambda function.
8. Create or update the S3 bucket policy to include the static IP address.

{
    "Version": "2012-10-17",
    "Id": "Policy1234567890",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<s3-bucket-name>/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        "<IP-Address>"
                    ]
                }
            }
        }
    ]
}

You should be able to access the object from the lambda using GET method.

Answer 2

short answer

Make sure the role you are using has the next policy and the role is attached to the lambda you are using:

{
  "Version": "version_id",
  "Statement": [
    {
        "Sid": "some_id",
        "Effect": "Allow",
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::bucketname",
            "arn:aws:s3:::bucketname/*"
        ]
    }
  ] 
}

For best practices only use the permissions you are supposed to

Long Answer

Create an IAM policy that defines the permissions for the Lambda function. The required permissions include:

• Get the object from the source S3 bucket.

• Put the resized object into the target S3 bucket.

• Permissions related to the CloudWatch Logs.

create an IAM Policy

Open the Policies page in the IAM Console.

Choose Create policy

Under the JSON tab, copy the following policy. Make sure the source and target bucket names match your bucketthat you created.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::bucketname/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::bucketname/*"
        },
        {
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket"
          ],
          "Resource": [
              "arn:aws:s3:::bucketname"
          ]
      }
    ]
}

• Choose Review policy, specify the policy name and create the policy.

Create the execution role

Create the execution role that gives your function permission to access AWS resources.

• Open the roles page in the IAM console.

• Choose Create role.

Create a role with the following properties.

Trusted entity – AWS Lambda.

Permissions – Choose the policy created earlier.

Role name – lambda-s3-role.

The AWSLambdaS3Policy policy has the permissions that the function needs to manage objects in Amazon S3 and if you haven't attached the role to lambda. you can attach it in the lambda configurations.

Answer 3

Your current Resource arn:aws:s3:::my-bucket-demo/* represents only objects in the my-bucket-demo. Subsequently, any action related to bucket (e.g. ListBucket) does not apply. You should add bucket resource ``arn:aws:s3:::my-bucket-demo` to your policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketTagging",
                "s3:GetInventoryConfiguration",
                "s3:GetObjectVersionTagging",
                "s3:ListBucketVersions",
                "s3:GetBucketLogging",
                "s3:ListBucket",
                "s3:GetAccelerateConfiguration",
                "s3:GetBucketPolicy",
                "s3:GetStorageLensConfigurationTagging",
                "s3:GetObjectVersionTorrent",
                "s3:GetObjectAcl",
                "s3:GetEncryptionConfiguration",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetIntelligentTieringConfiguration",
                "s3:GetBucketRequestPayment",
                "s3:GetAccessPointPolicyStatus",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectTagging",
                "s3:GetMetricsConfiguration",
                "s3:GetBucketOwnershipControls",
                "s3:GetBucketPublicAccessBlock",
                "s3:GetBucketPolicyStatus",
                "s3:ListBucketMultipartUploads",
                "s3:GetObjectRetention",
                "s3:GetBucketWebsite",
                "s3:GetJobTagging",
                "s3:GetBucketVersioning",
                "s3:GetBucketAcl",
                "s3:GetObjectLegalHold",
                "s3:GetBucketNotification",
                "s3:GetReplicationConfiguration",
                "s3:ListMultipartUploadParts",
                "s3:GetObject",
                "s3:GetStorageLensConfiguration",
                "s3:GetObjectTorrent",
                "s3:DescribeJob",
                "s3:GetBucketCORS",
                "s3:GetAnalyticsConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetBucketLocation",
                "s3:GetAccessPointPolicy",
                "s3:GetObjectVersion",
                "s3:GetStorageLensDashboard"
            ],
            "Resource": ["arn:aws:s3:::my-bucket-demo",
                         "arn:aws:s3:::my-bucket-demo/*"]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListStorageLensConfigurations",
                "s3:GetAccessPoint",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListAllMyBuckets",
                "s3:ListAccessPoints",
                "s3:ListJobs",
                "s3:ListObjects"
            ],
            "Resource": "*"
        }
    ]
}