Max
Max

Reputation: 13334

Serverless AWS Lambda@Edge: How to debug

I have a python Lambda@Edge function deployed with serverless which is working OK using the test feature:

enter image description here

FYI it's setup as viewer-request:

enter image description here

functions:
  cfLambda:
    handler: handler.lambda_handler
    events:
      - cloudFront:
          eventType: viewer-request

And if I go to CloudWatch, I can see the logs:

enter image description here

Now when I test with cURL it fails with a 503:

HTTP/2 503 
content-type: text/html
content-length: 1019
server: CloudFront
date: Mon, 05 Apr 2021 07:24:45 GMT
x-cache: LambdaExecutionError from cloudfront
via: 1.1 XXXXXXXXXXXXXXXXXX.cloudfront.net (CloudFront)
x-amz-cf-pop: AMS50-C1
x-amz-cf-id: 4vYpBnOGd6yfgowoSpiCyBkh5cbV1g3IJf1H2Eheln89MpEnScL-1g==

However this time I get no logs in CloudWatch. Q1: How can I have traces of my Lambda@Edge CloudFront calls visible in CloudWatch?

If I read the Lambda@Edge debug guide it says that 503 status code is either:

If I look at the console tests, they only consume ~220ms and ~75MB so I think we're way below the 5-second / 128MB limit for viewer request

enter image description here

If I look at the CloudFront logs they seem useless as they just confirm the 503:

E2HX7F6YEZN897.2021-04-04-16.a77a21e1:2021-04-04    16:34:12    SEA19-C3    389 35.247.33.169   HEAD    XXXXXXXX.cloudfront.net /   503 -   Mozilla/5.0%20(Windows%20NT%205.1)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/41.0.2224.3%20Safari/537.36  -   -   LambdaExecutionError    AcM5SX3ggB53fmjXO83xND_Lw3-eHXd8dlIZGEO53XaDMjuctRw==   example.org https   223 0.021   -   TLSv1.3 TLS_AES_128_GCM_SHA256  LambdaExecutionError    HTTP/1.1    -   -   51810   0.021   LambdaExecutionError    text/html   1019    -   -

Q2: Is there a way to increase the verbosity of the CloudFront logs (I couldn't find one)?

Q3: If I can't get my CloudFront Lambda@Edge calls in CloudWatch (Q1=no) and I can't increase verbosity of CloudFront Logs (Q2=no), how can I debug this further?

Upvotes: 3

Views: 2966

Answers (4)

DuyVinh
DuyVinh

Reputation: 457

I choose CloudFront with 'Use only North America and Europe', and I'm in Vietnam. Turn out the CloudWatch log was located in us-west-2 Oregon. That because the lambda function is trigger to edge location closest to me, which is somehow us-west-2. So just go to CloudWatch > Log groups, and change region until you see your log

Upvotes: 0

GSSwain
GSSwain

Reputation: 6133

  • The Lambda@Edge function must be deployed to the us-east-1 region.

  • The x-amz-cf-pop header gives a hint about where the request was executed. You can refer to this unofficial list here.

  • For us-east-1 the logs can be found in CloudWatch under the group /aws/lambda/<Your-function-name>. For any other region the log group would be /aws/lambda/us-east-1.<Your-function-name>. If you know the region, then select the appropriate region. Go to CloudWatch and search the appropriate log group. enter image description here

  • You can also navigate to the appropriate logs from the CloudFront page. Go to the Monitoring section -> Choose your Distribution -> View Distribution Metrics -> Lambda@Edge Errors. The graph would display the errors from all the regions when you hover over the data points. Once you know the region where the error is happening, you can select the same followed by the Lambda function and finally click on View logs. Refer to the below image. enter image description here

  • Take a look at some official examples here. For the viewer-request event, the ones manipulating the request are relevant.

  • The CloudFront logs can be accessed at /aws/cloudfront/LambdaEdge/<YourDistributionId>

Hopefully this helps you proceed further.

Upvotes: 4

Jordan
Jordan

Reputation: 2371

There's a few key considerations here.

Firstly, it's important to note how edge functions work.

Once deployed, your edge function is replicated across all AWS regions.

When a Cloudfront request comes in, it gets routed to the nearest available region to the user's location.

As a result, the function executes its logs in the region closest to the caller and not in the region of initial deployment.

Sometimes this may be a little bit counter-intuitive. It may be possible to be based in England, but be closer to the Ireland region for example.

Next, it's possible your lambda functions don't have the required permissions to log to Cloudfront.

When using Cloudfront logs you need to explicitly give your function logging permissions.

Consider the following IAM role:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": "logs:CreateLogGroup",
        "Resource": "arn:aws:logs:*:*:*"
    },
    {
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": [
            "arn:aws:logs:*:*:log-group:*:*"
        ]
    }
  ]
}

This role enables the function to create log groups, log streams and to put events into the logs for both Cloudfront and Cloudwatch.

Notice the wildcards, this is because when deploying to the edge you need to provide Cloudfront specific permission using the arn arn:aws:logs:*:*:log-group:/aws/cloudfront/*.

Upvotes: 1

gCoh
gCoh

Reputation: 3089

Have you tried AWS x-ray? https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html

It's a distributed tracing util lets you debug lambda functions. Probably will require some setup overhead, but in the long run might come in handy

Upvotes: -1

Related Questions