CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambdaboto3
KarloSpacapan
KarloSpacapan

Reputation: 373

504 timeout accessing S3 from Lambda with boto3

This is not an issue with the size of the files, the files are very small. It works perfectly where running locally, but not in the Lambda.

I attached a bucket-access policy to Lambda role mylambdarole: enter image description here

which looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket",
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

I also added a permission policy to my-bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Lambda permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::XXX843121XXX:role/mylambdarole"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my-bucket",
                "arn:aws:s3:::my-bucket/*"
            ]
        }
    ]
}

Lambda is in a VPC, but I think this shouldn't be a problem as I have a route to internet gateway:

enter image description here

The endpoint uses boto3, which for example loads data from the bucket:

def get_user_data_from_bucket(user_id: str):
    s3_client = boto3.client("s3", aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=REGION)
    try:
        data = s3_client.get_object(Bucket=BUCKET_NAME, Key=f"{user_id}/user_data.json")
        return json.loads(data["Body"].read())
    except ClientError as e:
        if e.response["Error"]["Code"] == "NoSuchKey":
            return {}
        raise e

I also tried adding config=Config(s3={'addressing_style':'path'}) to the client, but it didn't solve the issue.

As I said, it works when running the server locally. However, when the endpoint for downloading data is called, there is no error response, only after some time I get 504: Endpoint request timed out. Accessing RDS from the same Lambda function works just fine.

Upvotes: 0

Views: 1347

Answers (1)

jarmod
jarmod

Reputation: 78803

You configured your Lambda function to connect to a VPC public subnet. Lambda functions connected to a public subnet cannot access the internet. Even though it routes to the IGW, the Lambda does not have a public IP, and so its outbound packets are dropped at the IGW. More details here.

Re-configure the Lambda function to connect to a private subnet.

Also, there's no need for that S3 bucket policy (use the IAM role) or the manual credentials (use the IAM role).

Upvotes: 2

Related Questions