CloudFormation AutoscalingGroup "LoadBalancer attachments did not stabilize"

Question

So I'm using ECS (creation through ecs-cli) and CloudFormation, and I'm having an issue creating the autoscaling group:

It consistently fails saying "LoadBalancer attachments did not stabilize". Does anyone know what could be causing this??

I have two CloudFormation stacks, one main one which sets up most of my infrastructure, and the second one (which is failing) for a second ECS cluster. I'm passing input parameters from the outputs of the first/main stack.

I thought maybe it was a subnet size issue (They get created in the first stack and get passed to the second, 10.0.0.0/24 and 10.0.1.0/24), so I've tried creating two new subnets in the second cloudformation template and use them, but it resulted in the same error.

The Autoscaling group and ELB are being created the same between the two template files...

FIRST STACK:

"InternetGateway": {
        "Condition": "CreateVpcResources",
        "Type": "AWS::EC2::InternetGateway"
    },
    "AttachGateway": {
        "Condition": "CreateVpcResources",
        "Type": "AWS::EC2::VPCGatewayAttachment",
        "Properties": {
            "VpcId": {
                "Ref": "Vpc"
            },
            "InternetGatewayId": {
                "Ref": "InternetGateway"
            }
        }
    },
    "RouteViaIgw": {
        "Condition": "CreateVpcResources",
        "Type": "AWS::EC2::RouteTable",
        "Properties": {
            "VpcId": {
                "Ref": "Vpc"
            }
        }
    },
    "PublicRouteViaIgw": {
        "Condition": "CreateVpcResources",
        "DependsOn": "AttachGateway",
        "Type": "AWS::EC2::Route",
        "Properties": {
            "RouteTableId": {
                "Ref": "RouteViaIgw"
            },
            "DestinationCidrBlock": "0.0.0.0/0",
            "GatewayId": {
                "Ref": "InternetGateway"
            }
        }
    },
    "PubSubnet1RouteTableAssociation": {
        "Condition": "CreateVpcResources",
        "Type": "AWS::EC2::SubnetRouteTableAssociation",
        "Properties": {
            "SubnetId": {
                "Ref": "PubSubnetAz1"
            },
            "RouteTableId": {
                "Ref": "RouteViaIgw"
            }
        }
    },
    "PubSubnet2RouteTableAssociation": {
        "Condition": "CreateVpcResources",
        "Type": "AWS::EC2::SubnetRouteTableAssociation",
        "Properties": {
            "SubnetId": {
                "Ref": "PubSubnetAz2"
            },
            "RouteTableId": {
                "Ref": "RouteViaIgw"
            }
        }
    },
"Outputs": {
    "VpcId": {
        "Value": { "Ref": "Vpc" }
    },
    "KeyName": {
        "Value": { "Ref": "KeyName" }
    },
    "SourceCidr": {
        "Value": { "Ref": "SourceCidr"}
    },
    "EcsInstancePolicy": {
        "Value": { "Ref": "EcsInstancePolicy" }
    },
    "SubnetIds": {
        "Value": { 
            "Fn::Join": [
                ",", [{
                        "Ref": "PubSubnetAz1"
                    },
                    {
                        "Ref": "PubSubnetAz2"
                    }
                ]
            ]
        }
    },
    "CloudSecurityGroup": {
        "Value": { "Ref": "EcsSecurityGroup" }
    },
    "GatewayRouteTable": {
        "Value": { "Ref": "PublicRouteViaIgw" }
    }
}

SECOND STACK:

"Parameters": {
    "EcsAmiId": {
        "Type": "String",
        "Description": "ECS EC2 AMI id",
        "Default": ""
    },
    "EcsInstanceType": {
        "Type": "String",
        "Description": "ECS EC2 instance type",
        "ConstraintDescription": "must be a valid EC2 instance type."
    },
    "KeyName": {
        "Type": "AWS::EC2::KeyPair::KeyName",
        "Description": "Required - Name of an existing EC2 KeyPair to enable SSH access to the ECS instances"
    },
    "VpcId": {
        "Type": "String",
        "Description": "Required - VPC Id of existing VPC of Central stack.",
        "AllowedPattern": "^(?:vpc-[0-9a-f]{8}|)$",
        "ConstraintDescription": "VPC Id must begin with 'vpc-'"
    },
    "SubnetIds": {
        "Type": "String",
        "Description": "Required - Comma separated list of two (2) existing VPC Subnet Ids where ECS instances will run."
    },
    "AsgMaxSize": {
        "Type": "Number",
        "Description": "Maximum size and initial Desired Capacity of ECS Auto Scaling Group",
        "Default": "1"
    },
    "SourceCidr": {
        "Type": "String",
        "Description": "Required - Input CIDR/IP range to open up for ECS and Aurora"
    },
    "EcsInstancePolicy": {
        "Type": "String",
        "Description": "Required - IAM Policy for the ECS instances to use"
    },
    "EcsCluster": {
        "Type": "String",
        "Description": "ECS Cluster Name",
        "Default": "default"
    },
    "CloudSecurityGroup": {
        "Type": "String",
        "Description": "Name of the security group used by the ECS instances in the Cloud cluster"
    },
},
"Resources": {
    "EcsSecurityGroup": {
        "Type": "AWS::EC2::SecurityGroup",
        "Properties": {
            "GroupDescription": "ECS Allowed Ports",
            "VpcId": { "Ref": "VpcId" },
            "SecurityGroupIngress": [
                {
                    "IpProtocol": "tcp",
                    "FromPort": 22,
                    "ToPort": 22,
                    "SourceSecurityGroupId": { "Ref": "CloudSecurityGroup" }
                },
                {
                    "IpProtocol": "tcp",
                    "FromPort": 11000,
                    "ToPort": 11001,
                    "SourceSecurityGroupId": { "Ref": "CloudSecurityGroup" }
                },
                {
                    "IpProtocol": "tcp",
                    "FromPort": 22,
                    "ToPort": 22,
                    "CidrIp": { "Ref": "SourceCidr" }
                },
                {
                    "IpProtocol": "tcp",
                    "FromPort": 11000,
                    "ToPort": 11001,
                    "CidrIp": { "Ref": "SourceCidr" }
                }
            ]
        }
    },
    "EcsSecurityGroupIngressSelf": {
        "Type": "AWS::EC2::SecurityGroupIngress",
        "Properties": {
            "GroupId": { "Ref": "EcsSecurityGroup" },
            "SourceSecurityGroupId": { "Ref": "EcsSecurityGroup" },
            "IpProtocol": "tcp",
            "FromPort": 22,
            "ToPort": 9999
        }
    },
    "ElasticLoadBalancer": {
        "Type": "AWS::ElasticLoadBalancing::LoadBalancer",
        "Properties": {
            "Subnets": {
                "Fn::Split": [
                    ",",
                    { "Ref": "SubnetIds" }
                ]
            },
            "CrossZone": "true",
            "SecurityGroups": [{
                "Ref": "EcsSecurityGroup"
            }],
            "Listeners": [{
                    "LoadBalancerPort": "22",
                    "InstancePort": "22",
                    "Protocol": "TCP"
                },
                {
                    "LoadBalancerPort": "11000",
                    "InstancePort": "11000",
                    "Protocol": "TCP"
                },
                {
                    "LoadBalancerPort": "11001",
                    "InstancePort": "11001",
                    "Protocol": "TCP"
                }
            ],
            "HealthCheck": {
                "HealthyThreshold": "2",
                "Interval": "30",
                "Target": "TCP:22",
                "Timeout": "5",
                "UnhealthyThreshold": "5"
            }
        }
    },
    "EcsInstanceProfile": {
        "Type": "AWS::IAM::InstanceProfile",
        "Properties": {
            "Path": "/",
            "Roles": [{
                "Ref": "EcsInstancePolicy"
            }]
        }
    },
    "EcsInstanceLc": {
        "Type": "AWS::AutoScaling::LaunchConfiguration",
        "Properties": {
            "ImageId": {
                "Ref": "EcsAmiId"
            },
            "InstanceType": {
                "Ref": "EcsInstanceType"
            },
            "AssociatePublicIpAddress": true,
            "IamInstanceProfile": {
                "Ref": "EcsInstanceProfile"
            },
            "KeyName": {
                "Ref": "KeyName"
            },
            "SecurityGroups": [{
                "Ref": "EcsSecurityGroup"
            }],
            "UserData": {
                "Fn::Base64": {
                    "Fn::Join": [
                        "", [
                            "#!/bin/bash\n",
                            "echo ECS_CLUSTER=",
                            {
                                "Ref": "EcsCluster"
                            },
                            " >> /etc/ecs/ecs.config\n"
                        ]
                    ]
                }
            }
        }
    },
    "EcsInstanceAsg": {
        "Type": "AWS::AutoScaling::AutoScalingGroup",
        "Properties": {
            "VPCZoneIdentifier": [{ "Ref": "SubnetIds" }],
            "LaunchConfigurationName": {
                "Ref": "EcsInstanceLc"
            },
            "MinSize": "1",
            "MaxSize": {
                "Ref": "AsgMaxSize"
            },
            "DesiredCapacity": {
                "Ref": "AsgMaxSize"
            },
            "LoadBalancerNames": [{ "Ref": "ElasticLoadBalancer" }],
            "Tags": [{
                "Key": "Name",
                "Value": {
                    "Fn::Join": [
                        "", [
                            "ECS Instance - ",
                            {
                                "Ref": "AWS::StackName"
                            }
                        ]
                    ]
                },
                "PropagateAtLaunch": "true"
            }]
        }
    },

Please let me know if any extra information would be helpful

Answer 1

I got it to work. The solution was one (or both) of the following two changes I made:

1. I realized that I wasn't specifying the VPC for ecs-cli to start the cluster out in the existing VPC. I changed my code to include the '--vpc' and '--subnets' options in the 'ecs-cli up' call. This is probably the kicker, because as the documentation says, updating the VPCZoneIdentifier replaces the instances in the ASG, but not the ASG itself.

2. I removed the square brackets around the AutoScalingGroup's "VPCZoneIdentifier" parameter, so it now looks like this: "VPCZoneIdentifier": { "Ref": "SubnetIds" }, where SubnetIds is a String of two subnet ids with a comma between them. (Note that as of 2/7/2017, The documentation is wrong. It says that this parameter takes a list of Strings, but it obviously doesn't.)

Answer 2

From your screenshot, it appears that the CloudFormation stack along with the EcsInstanceAsg Auto Scaling Group were previously created, and you're attempting to update the Auto Scaling Group to reference a newly-created Load Balancer.

The most common issue with CloudFormation resources failing to stabilize on updates is due to referenced resources being modified and/or deleted outside of the CloudFormation stack. This causes CloudFormation to modify a resource that it can no longer find, which can cause random errors or timeouts, and is discouraged according to AWS CloudFormation Best Practices. If this is the case, the best way to proceed is to make a fresh start with a brand new stack, if possible.

If this is not the case for you, there might be an unknown limitation or issue with in-place updates to the LoadBalancerNames property in the AWS::AutoScaling::AutoScalingGroup (support for in-place updates to this property was only just added on Jan 17 2017, so there might still be issues with it). Try re-creating your Auto Scaling Group (changing the name of the EcsInstanceAsg resource in your template would cause it to be re-created) and see if that solves the issue.