Reading CSV file from S3 using Lambda Function-GetObject operation: Access Denied

Question

I am now trying to read CSV file(only column name) from S3 bucket using Lambda function. I have created an S3 trigger within Lambda. Here is the sample code;

import json
import boto3
import csv

s3_client = boto3.client('s3')
def lambda_handler(event, context):
    # TODO implement
    bucket = event['Records'][0]['s3']['bucket']['name']
    csv_file = event['Records'][0]['s3']['object']['key']
    response = s3_client.get_object(Bucket=bucket, Key=csv_file)
    lines = response['Body'].read().decode('utf-8').split()
    results = []
    for row in csv.DictReader(lines):
        results.append(row.name())
    print(results)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Whenever I try to upload a new file, i get this error;

[ERROR] ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 10, in lambda_handler
    response = s3_client.get_object(Bucket=bucket, Key=csv_file)
  File "/var/runtime/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name

I added a specific role and provided necessary permissions to my S3 bucket.

Snippet of my S3 bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::my-bucket/*",
}
}

I have provided necessary permissions to my s3 bucket. But still getting this error:

Response
{
  "errorMessage": "An error occurred (AccessDenied) when calling the GetObject operation: Access Denied",
  "errorType": "ClientError",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 19, in lambda_handler\n    response = s3_client.get_object(Bucket=bucket, Key=csv_file)\n",
    "  File \"/var/runtime/botocore/client.py\", line 386, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
    "  File \"/var/runtime/botocore/client.py\", line 705, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
  ]
}

Can anyone tell me why am i getting this error?

Answer 1

Assuming that your S3 bucket is the one in-charge of invoking the lambda function. This will require two parties to have permissions.

1). The bucket needs to have a policy that allows it to trigger the function. 2). The lambda that will pull the CSV files from the bucket needs policy too man. In order to achieve the second part, you might want to consider pre-built policy templates available in SAM templates, this will not only make your policy definition more readable but also limits the actions that your lambda can perform on your buckets. The first sample below showcases how to grant S3 CRUD permissions

S3CsvReactor:
  Type: "AWS::Serverless::Function"
  Name: "csv-process-function"
  Properties:
    CodeUri: csv-processor-function/
    Handler: app.execute
    Timeout: 30 # Seconds
    Runtime: Python 3.8
    MemorySize: 512
    Policies:
      - S3CrudPolicy:
          BucketName: "s3-containing-your-csv"

This example below showcases read-only implementation

S3CsvReactor:
  Type: "AWS::Serverless::Function"
  Name: "csv-process-function"
  Properties:
    CodeUri: csv-processor-function/
    Handler: app.execute
    Timeout: 30 # Seconds
    Runtime: Python 3.8
    MemorySize: 512
    Policies:
      - S3ReadPolicy:
          BucketName: "s3-containing-your-csv"

This example below showcases write-only implementation

S3CsvReactor:
  Type: "AWS::Serverless::Function"
  Name: "csv-process-function"
  Properties:
    CodeUri: csv-processor-function/
    Handler: app.execute
    Timeout: 30 # Seconds
    Runtime: Python 3.8
    MemorySize: 512
    Policies:
      - S3WritePolicy:
          BucketName: "s3-containing-your-csv"

Answer 2

Make sure the role can perform the PutObject and GetObject IAM actions on the bucket in the IAM resource specified in the IAM policy. Wrap your logic in a "try/except" block as well as a good practice to catch your errors, you might be surprised that the nature of the error from S3 that is propagating occurs earlier than expected. Furthermore, in the Lambda console select this function then click on the "Monitor" tab to be redirected to the CloudWatch Logs console, then you can read more details about the error especially if you are returning exceptions early.