Application Load Balancer's DNS is timing out eventhough the Target group is in healthy state

Question

I changed the status code for the health check from 200 to 302. After which the Target group turned to a healthy state. I wasn't able to get the healthy state with the 200 code. But when I try to access the DNS of the ALB. It times out and haven't been able to figure out why?

The ecs drupal instances logs provide these outputs "GET / HTTP/1.1" 302 573 "-" "ELB-HealthChecker/2.0" drupal

Any help would be much appreciated

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  KeyName:
    Type: AWS::EC2::KeyPair::KeyName
    Description: Name of an existing EC2 KeyPair to enable SSH access to the ECS instances.
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: Select a VPC that allows instances access to the Internet.
  SubnetId:
    Type: List<AWS::EC2::Subnet::Id>
    Description: Select at two subnets in your selected VPC.
  DesiredCapacity:
    Type: Number
    Default: '1'
    Description: Number of instances to launch in your ECS cluster.
  MaxSize:
    Type: Number
    Default: '1'
    Description: Maximum number of instances that can be launched in your ECS cluster.
  InstanceType:
    Description: EC2 instance type
    Type: String
    Default: t2.medium
    AllowedValues:
      - t2.micro
      - t2.small
      - t2.medium
      - t2.large
      - m3.medium
      - m3.large
      - m3.xlarge
      - m3.2xlarge
      - m4.large
      - m4.xlarge
      - m4.2xlarge
      - m4.4xlarge
      - m4.10xlarge
      - c4.large
      - c4.xlarge
      - c4.2xlarge
      - c4.4xlarge
      - c4.8xlarge
      - c3.large
      - c3.xlarge
      - c3.2xlarge
      - c3.4xlarge
      - c3.8xlarge
      - r3.large
      - r3.xlarge
      - r3.2xlarge
      - r3.4xlarge
      - r3.8xlarge
      - i2.xlarge
      - i2.2xlarge
      - i2.4xlarge
      - i2.8xlarge
    ConstraintDescription: Please choose a valid instance type.

Mappings:
  AWSRegionToAMI:
    us-east-1:
      AMIID: ami-0be13a99cd970f6a9
    us-east-2:
      AMIID: ami-0a9e12068cb98a01d
    us-west-1:
      AMIID: ami-0fa6c8d131a220017
    us-west-2:
      AMIID: ami-078c97cf1cefd1b38
    eu-west-1:
      AMIID: ami-0c9ef930279337028
    eu-central-1:
      AMIID: ami-065c1e34da68f2b02
    ap-northeast-1:
      AMIID: ami-02265963d1614d04d
    ap-southeast-1:
      AMIID: ami-0b68661b29b9e058c
    ap-southeast-2:
      AMIID: ami-00e4b147599c13588
    ap-south-1:
      AMIID: ami-036eaa870decb368d
Resources:
  ECSCluster:
    Type: AWS::ECS::Cluster
    #DependsOn: ECSAutoScalingGroup
    #Properties:
      #CapacityProviders:
        #- Nse
  ECSCapacityProvider:
    Type: AWS::ECS::CapacityProvider
    Properties:
      AutoScalingGroupProvider:
        AutoScalingGroupArn: !Ref ECSAutoScalingGroup
      Name: Nse
  EcsSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: ECS Security Group
      VpcId: !Ref 'VpcId'
  EcsSecurityGroupHTTPinbound:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref 'EcsSecurityGroup'
      IpProtocol: tcp
      FromPort: '80'
      ToPort: '80'
      CidrIp: 0.0.0.0/0
  EcsSecurityGroupDrupalinbound:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref 'EcsSecurityGroup'
      IpProtocol: tcp
      FromPort: '8080'
      ToPort: '8080'
      CidrIp: 0.0.0.0/0
  EcsSecurityGroupSQLInbound:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref 'EcsSecurityGroup'
      IpProtocol: tcp
      FromPort: '3306'
      ToPort: '3306'
      CidrIp: 0.0.0.0/0
  EcsSecurityGroupSSHinbound:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref 'EcsSecurityGroup'
      IpProtocol: tcp
      FromPort: '22'
      ToPort: '22'
      CidrIp: 0.0.0.0/0
  EcsSecurityGroupALBports:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref 'EcsSecurityGroup'
      IpProtocol: tcp
      FromPort: '31000'
      ToPort: '61000'
      SourceSecurityGroupId: !Ref 'EcsSecurityGroup'
  
  CloudwatchLogsGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Join ['-', [ECSLogGroup, !Ref 'AWS::StackName']]
      RetentionInDays: 14 
  taskdefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Join ['', [!Ref 'AWS::StackName', -ecs-demo-app]]
      ExecutionRoleArn: arn:aws:iam::268500393272:role/ecsTaskExecutionRole
      #TaskRoleArn: arn:aws:iam::aws:policy/AmazonS3FullAccess
      NetworkMode: bridge
      #RequiredCompatibilities:
        #- "EC2"
      ContainerDefinitions:
        - Name: drupal
          Cpu: 256
          #Essential: false
          Image: drupal:latest
          Memory: 512
          Hostname: drupal
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref 'CloudwatchLogsGroup'
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: ecs-demo-app
          MountPoints:
            - ContainerPath: /var/www/html
              SourceVolume: drupal-data
          PortMappings:
            - ContainerPort: 80
              HostPort: 8080
      Volumes:
        - Name: drupal-data
  ECSALB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    #DependsOn: service
    Properties:
      Name: ECSALB
      Scheme: internet-facing
      LoadBalancerAttributes:
      - Key: idle_timeout.timeout_seconds
        Value: '60'
      Subnets: !Ref 'SubnetId'
      SecurityGroups: [!Ref 'EcsSecurityGroup']
  ALBListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn: ECSServiceRole
    Properties:
      DefaultActions:
      - Type: forward
        TargetGroupArn: !Ref 'ECSTG'
        #RedirectConfig:
          #Protocol: "HTTP"
          #Host: "#{host}"
          #Path: "/#{path}"
          #Query: "#{query}"
          #Port: 80
          #StatusCode: "HTTP_302"
      LoadBalancerArn: !Ref 'ECSALB'
      Port: '80'
      Protocol: HTTP
  ECSALBListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    DependsOn: ALBListener
    Properties:
      Actions:
      - Type: forward
        TargetGroupArn: !Ref 'ECSTG'
        #RedirectConfig:
          #Protocol: "HTTP"
          #Host: "#{host}"
          #Path: "/#{path}"
          #Query: "#{query}"
          #Port: 80
          #StatusCode: "HTTP_302"
      Conditions:
      - Field: path-pattern
        Values: [/]
      ListenerArn: !Ref 'ALBListener'
      Priority: 1
  ECSTG:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    DependsOn: ECSALB
    Properties:
      HealthCheckIntervalSeconds: 80
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 60
      HealthyThresholdCount: 2
      Matcher:
        HttpCode: 302
      Name: ECSTG
      Port: 8080
      Protocol: HTTP
      TargetType: instance
      UnhealthyThresholdCount: 2
      VpcId: !Ref 'VpcId'
  ECSAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      VPCZoneIdentifier: !Ref 'SubnetId'
      #VPCZoneIdentifier:
        #- subnet-0c228c2e5e42708aa
        #- subnet-0bf3fcea01d2dd0a4
        #- subnet-0c6a01197480771b3
      LaunchConfigurationName: !Ref 'ContainerInstances'
      #LoadBalancerNames:
        #- ECSALB
      TargetGroupARNs: [!Ref 'ECSTG']
      MinSize: '1'
      MaxSize: !Ref 'MaxSize'
      DesiredCapacity: !Ref 'DesiredCapacity'
    CreationPolicy:
      ResourceSignal:
        Timeout: PT15M
    UpdatePolicy:
      AutoScalingReplacingUpdate:
        WillReplace: 'true'
  ContainerInstances:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      ImageId: !FindInMap [AWSRegionToAMI, !Ref 'AWS::Region', AMIID]
      SecurityGroups: [!Ref 'EcsSecurityGroup']
      InstanceType: !Ref 'InstanceType'
      IamInstanceProfile: !Ref 'EC2InstanceProfile'
      KeyName: !Ref 'KeyName'
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash -xe
          echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
          yum update -y
          yum install tmux -y
          yum install -y aws-cfn-bootstrap
          #/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource ECSAutoScalingGroup --region ${AWS::Region}
          #sudo cat /var/log/cloud-init-output.log
          /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ECSAutoScalingGroup --region ${AWS::Region}
  service:
    Type: AWS::ECS::Service
    DependsOn: 
      - ALBListener
      #- ECSAutoScalingGroup
    Properties:
      Cluster: !Ref 'ECSCluster'
      DesiredCount: '1'
      HealthCheckGracePeriodSeconds: 2147483647
      LoadBalancers:
      - ContainerName: drupal
        ContainerPort: '80'
        TargetGroupArn: !Ref 'ECSTG'
        #LoadBalancerName: !GetAtt ECSALB.LoadBalancerName
        #LoadBalancerName: ECSALB
      #NetworkConfiguration:
      #  AwsvpcConfiguration:
      #    AssignPublicIp: ENABLED
      #    SecurityGroups:
      #      - !Ref 'EcsSecurityGroup'
      #    Subnets: !Ref 'SubnetId'
      #ServiceName: Ecs
      Role: !Ref 'ECSServiceRole'
      TaskDefinition: !Ref 'taskdefinition'
  ECSServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [ecs.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies:
      - PolicyName: ecs-service
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action: ['elasticloadbalancing:DeregisterInstancesFromLoadBalancer', 'elasticloadbalancing:DeregisterTargets',
              'elasticloadbalancing:Describe*', 'elasticloadbalancing:RegisterInstancesWithLoadBalancer',
              'elasticloadbalancing:RegisterTargets', 'ec2:Describe*', 'ec2:AuthorizeSecurityGroupIngress']
            Resource: '*'
  ServiceScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    #DependsOn: service
    Properties:
      MaxCapacity: 3
      MinCapacity: 1
      ResourceId: !Join ['', [service/, !Ref 'ECSCluster', /, !GetAtt [service, Name]]]
      RoleARN: !GetAtt [AutoscalingRole, Arn]
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
  ServiceScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: AStepPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref 'ServiceScalingTarget'
      StepScalingPolicyConfiguration:
        AdjustmentType: PercentChangeInCapacity
        Cooldown: 60
        MetricAggregationType: Average
        StepAdjustments:
        - MetricIntervalLowerBound: 0
          ScalingAdjustment: 200
  ALB500sAlarmScaleUp:
    Type: AWS::CloudWatch::Alarm
    Properties:
      EvaluationPeriods: '1'
      Statistic: Average
      Threshold: '10'
      AlarmDescription: Alarm if our ALB generates too many HTTP 500s.
      Period: '60'
      AlarmActions: [!Ref 'ServiceScalingPolicy']
      Namespace: AWS/ApplicationELB
      Dimensions:
        - Name: LoadBalancer
          Value: !GetAtt 
            - ECSALB
            - LoadBalancerFullName
      ComparisonOperator: GreaterThanThreshold
      MetricName: HTTPCode_ELB_5XX_Count
  EC2Role:
    Type: AWS::IAM::Role
    Properties:
      #ManagedPolicyArns:
        #- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [ec2.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies:
      - PolicyName: ecs-service
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action: ['ecs:CreateCluster', 'ecs:DeregisterContainerInstance', 'ecs:DiscoverPollEndpoint',
              'ecs:Poll', 'ecs:RegisterContainerInstance', 'ecs:StartTelemetrySession',
              'ecs:Submit*', 'logs:CreateLogStream', 'logs:PutLogEvents']
            Resource: '*'
  AutoscalingRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [application-autoscaling.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies:
      - PolicyName: service-autoscaling
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action: ['application-autoscaling:*', 'cloudwatch:DescribeAlarms', 'cloudwatch:PutMetricAlarm',
              'ecs:DescribeServices', 'ecs:UpdateService']
            Resource: '*'
  EC2InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles: [!Ref 'EC2Role']
Outputs:
  ecsservice:
    Value: !Ref 'service'
  ecscluster:
    Value: !Ref 'ECSCluster'
  ECSALB:
    Description: Your ALB DNS URL
    Value: !Join ['', [!GetAtt [ECSALB, DNSName]]]
  taskdef:
    Value: !Ref 'taskdefinition'

Answer 1

Based on the comments.

The template is fine. The ALB does not work because it is placed in private subnets along with ECS service. Assuming that private subnets are correctly setup to work with NAT gateway and access the internet, the following should be made:

• Place ALB in public subnets - it must be there, as otherwise it will no be accessible from the internet.

Also double check all the route tables for NAT, public subnets, internet gateway.