CODEWITHSUNDEEP
javascriptamazon-web-servicesaws-cloudformationaws-samaws-cloudformation-custom-resource
Maurdekye
Maurdekye

Reputation: 3697

AWS Custom Resource not recognizing success or failure signal

I'm trying to set up a custom resource in my stack to implement some database initialization steps during stack creation. However, while the custom resource code appears to run fine, CloudFormation doesn't recognize the SUCCESS signal sent back by the custom resource, and it gets stuck in the CREATE_IN_PROGRESS state for the full 1-hour timeout. Then, of course, when I try to manually delete the stack, it gets stuck for another hour in DELETE_IN_PROGRESS waiting for the delete signal.

I've coded a minimal recreation of my environment that exhibits the same behavior. Here is my lambda code:

const { put } = require('axios');

const SUCCESS = 'SUCCESS';
const FAILED = 'FAILED';

// CloudFormation Custom Resource Response Handler
let Responder = (event, context) => async (status, data={}) => {
    let body = {
        Status: status,
        Reason: data.Reason ?? data.Error ?? "",
        PhysicalResourceId: `${event.LogicalResourceId}-${event.RequestId}`,
        StackId: event.StackId,
        RequestId: event.RequestId,
        LogicalResourceId: event.LogicalResourceId,
        Data: (({Reason, Error, ...etc}) => etc)(data)
    };
    console.log("Custom Resource body:", body);
    let request = put(event.ResponseURL, body);
    console.log("Response sent");
    context.done();
    console.log("Context discontinued");
    let response = await request;
    console.log("Custom Resource response:", response);
}

exports.lambdaHandler = async (event, context) => {
    console.info("event:", event);

    let responder = Responder(event, context);
    try {
        if (event.RequestType === "Delete") {
            console.log("Custom resource deleting!");
            // delete resource
            await responder(SUCCESS);
        } else {
            console.log("Custom resource creating!");
            // create resource
            await responder(SUCCESS);
        }
    } catch (err) {
        console.error(err);
        await responder(FAILED, err);
    }
};

and here is my SAM template:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:

  RDSVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.10.0.0/16
      EnableDnsSupport: true
      EnableDnsHostnames: true

  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref RDSVPC
      CidrBlock: 10.10.1.0/24
      AvailabilityZone: !Select 
        - 0
        - !GetAZs 
          Ref: 'AWS::Region'
  
  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref RDSVPC
      CidrBlock: 10.10.2.0/24
      AvailabilityZone: !Select 
        - 1
        - !GetAZs 
          Ref: 'AWS::Region'

  InternetGateway:
    Type: AWS::EC2::InternetGateway
  
  InternetGatewayAttachment:
    DependsOn: InternetGateway
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref RDSVPC
      InternetGatewayId: !Ref InternetGateway

  RouteTable:
    DependsOn: InternetGatewayAttachment
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref RDSVPC
  
  RouteTableAttachement:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet1
      RouteTableId: !Ref RouteTable

  RouteTableAttachement2:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet2
      RouteTableId: !Ref RouteTable

  EIP:
    DependsOn: InternetGatewayAttachment
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc

  NAT:
    DependsOn: RDSVPC
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt EIP.AllocationId
      SubnetId: !Ref InternetSubnet
      ConnectivityType: public

  NATRoute:
    DependsOn: 
      - NAT
      - RouteTable
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref RouteTable
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NAT

  InternetSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref RDSVPC
      CidrBlock: 10.10.3.0/24
      AvailabilityZone: !Select 
        - 2
        - !GetAZs 
          Ref: 'AWS::Region'
  
  InternetRouteTable:
    DependsOn: InternetGatewayAttachment
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref RDSVPC
  
  InternetRouteTableAttachement:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref InternetSubnet
      RouteTableId: !Ref InternetRouteTable

  InternetRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref InternetRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Lambda SG
      VpcId: !Ref RDSVPC
      SecurityGroupEgress: 
        - CidrIp: "0.0.0.0/0"
          IpProtocol: "-1"

  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Timeout: 3
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs14.x
      Architectures:
        - x86_64
      VpcConfig:
        SecurityGroupIds:
          - !Ref LambdaSecurityGroup
        SubnetIds:
          - !Ref PublicSubnet1
          - !Ref PublicSubnet2

  HelloWorldCustomResource:
    DependsOn: 
      - HelloWorldFunction
      - InternetRoute
      - NATRoute
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !GetAtt HelloWorldFunction.Arn

Sample CloudWatch Log output (anonymized):

START RequestId: b7f45954-f5e9-4dd3-8404-c2117c8e8718 Version: $LATEST
2021-12-22T18:22:52.726Z    b7f45954-f5e9-4dd3-8404-c2117c8e8718    INFO    event: {
  RequestType: 'Create',
  ServiceToken: 'arn:aws:lambda:us-east-2:<userid>:function:custom-resource-test-2-HelloWorldFunction-DZYI8GLlnI0j',
  ResponseURL: 'https://cloudformation-custom-resource-response-useast2.s3.us-east-2.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-2%3A<userid>%3Astack/custom-resource-test-2/1820acc0-6354-11ec-b638-061066120c32%7CHelloWorldCustomResource%7C1311ae33-9ac4-461c-a118-433bc29e1671?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211222T182252Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIAVRFIPK6PKAFALJ4V%2F20211222%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=0190bb01d874ce33cf7d8559c060df5ffb171053b95279e9318cb556ff0f91e0',
  StackId: 'arn:aws:cloudformation:us-east-2:<userid>:stack/custom-resource-test-2/1820acc0-6354-11ec-b638-061066120c32',
  RequestId: '1311ae33-9ac4-461c-a118-433bc29e1671',
  LogicalResourceId: 'HelloWorldCustomResource',
  ResourceType: 'AWS::CloudFormation::CustomResource',
  ResourceProperties: {
    ServiceToken: 'arn:aws:lambda:us-east-2:<userid>:function:custom-resource-test-2-HelloWorldFunction-DZYI8GLlnI0j'
  }
}
2021-12-22T18:22:52.726Z    b7f45954-f5e9-4dd3-8404-c2117c8e8718    INFO    Custom resource creating!
2021-12-22T18:22:52.726Z    b7f45954-f5e9-4dd3-8404-c2117c8e8718    INFO    Custom Resource body: {
  Status: 'SUCCESS',
  Reason: '',
  PhysicalResourceId: 'HelloWorldCustomResource-1311ae33-9ac4-461c-a118-433bc29e1671',
  StackId: 'arn:aws:cloudformation:us-east-2:<userid>:stack/custom-resource-test-2/1820acc0-6354-11ec-b638-061066120c32',
  RequestId: '1311ae33-9ac4-461c-a118-433bc29e1671',
  LogicalResourceId: 'HelloWorldCustomResource',
  Data: {}
}
2021-12-22T18:22:52.996Z    b7f45954-f5e9-4dd3-8404-c2117c8e8718    INFO    Response sent
2021-12-22T18:22:53.016Z    b7f45954-f5e9-4dd3-8404-c2117c8e8718    INFO    Context discontinued
END RequestId: b7f45954-f5e9-4dd3-8404-c2117c8e8718
REPORT RequestId: b7f45954-f5e9-4dd3-8404-c2117c8e8718  Duration: 314.99 ms Billed Duration: 315 ms Memory Size: 128 MB Max Memory Used: 63 MB  Init Duration: 195.58 ms

As you can see, every resource created successfully, except for the custom resource, which is stuck: cloudformation stack output

Update: On adding full ingress allow rules to the lambda's security group...

  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Lambda SG
      VpcId: !Ref RDSVPC
      SecurityGroupEgress: 
        - CidrIp: "0.0.0.0/0"
          IpProtocol: "-1"
      SecurityGroupIngress:
        - CidrIp: "0.0.0.0/0"
          IpProtocol: "-1"

...the put request succeeds, and the custom resource creates successfully. However, this isn't a real solution; I can't simply allow full ingress on my lambda to the internet just to fix this issue. Is there a more specific ingress rule I can set to allow communication to work properly?

Upvotes: 1

Views: 1914

Answers (1)

BlueSky
BlueSky

Reputation: 1

I think event.ResponseURL is an object and it contains some items. You can use the href filed in the event.ResponseURL. Or you can use postname and path fields, etc.

Of course you can check the cfn-response source code and base on it you implement your own source code.

Upvotes: 0

Related Questions