CODEWITHSUNDEEP
amazon-web-servicesaws-cloudformationserverlesscircleciinfrastructure-as-code
Peter Bailey
Peter Bailey

Reputation: 105898

How to debug CloudFormation templates generated by Serverless orchestrated by CircleCI

Context

I feel like I need to provide a lot of context here for the full scope of the problem to be understood, so apologies in advance if this goes a bit long or ends up providing too much information, I just want to ward off as many follow-up questions and clarification requests as I can.

I've got a project that's a tech handoff. It used to be in production under a different owner in a different AWS account. I'm trying to re-launch it in an AWS account I control and one of the packages is creating some problems for me.

It uses Serverless to provision a couple S3 buckets and their access policies, a couple IAM roles, and a bunch of ApiGateway methods. The package relies on nested stacks to get around the 200 resource limit as described here.

Finally, the IAM user that CircleCI connects as has the AdministratorAccess policy attached.

Problem

I keep getting failures from CircleCI during this step in the build

node_modules/.bin/serverless deploy --verbose --stage develop --region us-east-1 --package ./.serverless

The exact nature of the failure seems to be inconsistent i.e., it doesn't always fail at the same spot. At some point a resource just fails to create and the whole process rolls back. Here are a couple examples of run failures in the log with +/- 5 lines, followed by the actual error reported by Serverless

Run 1

CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod002VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod003VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod004VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod006Options
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncAbcNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncDefNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncGhiNestedStack
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethod006Options

  Serverless Error ---------------------------------------

  An error occurred: FuncAbcNestedStack - Embedded stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/org-package-develop-FuncAbcNestedStack/RESOURCE-ID-001 was not successfully created: The following resource(s) failed to create: [AbcLambdaFunction]. .

Run 2

CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod002VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod005VarOptions
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod006Options
CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethod004VarOptions
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncDefNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncGhiNestedStack
CloudFormation - CREATE_FAILED - AWS::CloudFormation::Stack - FuncAbcNestedStack
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - org-package-develop
CloudFormation - DELETE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethod001VarOptions

  Serverless Error ---------------------------------------

  An error occurred: FuncDefNestedStack - Embedded stack arn:aws:cloudformation:us-east-1:ACCOUNT_ID:stack/org-package-develop-FuncDefNestedStack/RESOURCE-ID-002 was not successfully created: The following resource(s) failed to create: [DefLambdaFunction]. .

Note: All the unique identifiers in the above logs have been replaces/obfuscated by new identifiers which are unique across both logs, not per log i.e., FuncAbcNestedStack appears in both logs because it exact same resource in the configuration.

Question

Given all the above, my question at this point is how do I debug this? This represents all the detail I (believe) is available to me in that I can't dive deeper to find out why a resource failed to create. I've read a bit about troubleshooting errors but nothing there has been terribly helpful since I'm not actually using EC2 directly.

Apr 4 Update

I've done a good amount of work trying to debug the templates. Mind you, I'm generally not working with the templates themselves, Serverless generates them and dumps them into an S3 bucket before they're applied.

Here are some steps I've taken

  1. Updated to most recent version of Serverless (1.67.0, from 1.30.3)
  2. Nuked existing stacks
  3. Nuked related S3 bucket
  4. Updated node runtime (12.16.1, from 8.10.0)
  5. Downloaded and linted the CFN template that contains the failing lambda - no issues reported

I'm still getting the same results. When I re-run the build and check the CloudFormation event logs, I do see that a stack fails to create because a Lambda function within it fails to create. There's nothing special about this function (other Lambdas create successfully earlier in the run) other than the fact that it's the authorizer for every other function in the API, which may or may not be significant. I still can't find further detail as to why the lambda fails to create.

Apr 6 Update

Ok, now that I understand how the CloudFormation console works, here is now what I think is the undermost-lying error message

Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 0507722d-46e7-4340-bc68-fdba1ed469d6)

Looking at the CFN template itself for this nested stack, I now see what is going on. Every single lamba gets its own stack, and each stack across the entire package is compiled into a single ZIP file, whose size ends up being about 270MB or 20MB larger than the limit specified above. From this point, it seems like I have two possible paths forward

  1. Figure out how to split the functions across multiple ZIPs
  2. Change the webpack configuration so the compiled files are less bloated (I seriously don't know what's going on here - a 1k TypeScript file is coming out as 6.5MB after webpack)

Upvotes: 5

Views: 1176

Answers (1)

Pat Myron
Pat Myron

Reputation: 4638

You'll need to look at the nested stacks themselves. The AbcLambdaFunction and DefLambdaFunction resources should have more detailed failure stack events in the nested stacks than in the parent stacks. You'll likely need to fix AbcLambdaFunction and DefLambdaFunction in the nested stack templates, as the inconsistency is likely just due to whichever resource happened to fail first and started the rollback

If it's been a while since those templates have been run, it's likely Lambda Runtimes have been deprecated. The CloudFormation Linter should be able to check your templates for this and more possibilities

AWS Lambda limits are likely as well, I'd recommend trying things like this

Check to see if there any CloudWatch logs as well

Upvotes: 2

Related Questions