AWS HTTP API Gateway with custom link to private ALB

Question

I have the following setup:

Custom Domain api.foo.co.uk-> API Mapping to stage v1 -> HTTP API path ANY /{proxy+} -> Private VPC Link -> ALB Fargate

If I hit the Custom Domain api.foo.co.uk I get a 503 "message": "Service Unavailable" If I hit the API direct p3dqjsdfszlv7.execute-api.eu-west-1.amazonaws.com/v1/ I get the same

In the CW for the API I see the following:

{
"auth_status":"-",
"aws_endpoint":"-",
"cognito_auth_provider":"-",
"cognito_auth_type":"-",
"cognito_identity_id":"-",
"cognito_identity_pool_id":"-",
"domain_name":"api.foo.co.uk",
"domain_prefix":"api",
"err_msg":"Service Unavailable",
"err_response":"INTEGRATION_NETWORK_FAILURE",
"err_string":" "Service Unavailable"",
"http_method":"GET",
"integration_error":"-",
"integration_error_msg":"-",
"integration_int_status":"200",
"integration_status":"-",
"path":"/v1/",
"principa_ord_id":"-",
"protocol":"HTTP/1.1",
"request_id":"cSJJ2h7BjoEEJ-g=",
"route_key":"ANY /{proxy+}",
"source_ip":"22.22.103.68",
"stage":"v1",
"status":"503",
"time":"16/Mar/2021:14:08:24 +0000",
"user":"-",
"user_agent":"insomnia/2021.1.0"}

Anyone have any idea what the issue might be ? I have looked and can not find anything on the error message err_response":"INTEGRATION_NETWORK_FAILURE" or what the cause might be.

I have also enabled the access logs on the ALB but they are blank so I am assuming its not getting as far as the ALB

Answer 1

This might be the root cause of the 503 errors. VPC links require to use private subnets. If I put public subnets along with the private subnets it returns 503 errors most of the time. After removing public subnets from the VPC Link it is fixed.

https://repost.aws/questions/QUR19Keq0OQ_qPin1MOBbvzA/http-api-alb-integration-5xx-errors

Answer 2

I also had the INTEGRATION_NETWORK_FAILURE error;

I created my ALB with the default scheme type (Internet-facing) instead of Internal. So changing the scheme did the job (seems obvious now :-) )

Answer 3

I had the same problem of INTEGRATION_NETWORK_FAILURE. I managed to find a more informative error message by including all of the $context.integration* variables in the access logs (see https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-logging-variables.html). In my case there was an error message that said "Request failed due to a network error communicating with the endpoint".

I can't tell what was the exact cause or what made it go away, but I can share some CloudFormation snippets of the final working setup:

1. ALB setup - listening for HTTP requests on port 80:

  ApplicationLoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub "${EnvironmentName} ALB security group"
      GroupName: !Sub "${EnvironmentName}-load-balancer-sg"
      VpcId:
        'Fn::ImportValue': !Sub "${EnvironmentName}:VPC"
      SecurityGroupIngress:
        - CidrIp: "0.0.0.0/0"
          IpProtocol: "tcp"
          FromPort: 80
          ToPort: 80
      SecurityGroupEgress:
        - CidrIp: "0.0.0.0/0"
          IpProtocol: "-1"

  SharedApplicationLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Sub "${EnvironmentName}-shared-lb"
      Scheme: "internal"
      Type: "application"
      Subnets:
        - 'Fn::ImportValue': !Sub "${EnvironmentName}:${LBSubnetType}1"
        - 'Fn::ImportValue': !Sub "${EnvironmentName}:${LBSubnetType}2"
      SecurityGroups:
        - !Ref ApplicationLoadBalancerSecurityGroup
      IpAddressType: "ipv4"

  SharedApplicationLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
      - SharedApplicationLoadBalancer
    Properties:
      LoadBalancerArn: !Ref SharedApplicationLoadBalancer
      Protocol: "HTTP"
      Port: 80
      DefaultActions:
        - Type: fixed-response
          FixedResponseConfig:
            StatusCode: 404
            MessageBody: Shared ALB has no such route
            ContentType: text/plain

2. Per-service ALB target group:

  SharedAlbServiceXTargetGroup:
    Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
    Properties:
      Name: !Sub "${EnvironmentName}-alb-${ServiceName}-tg"
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: "/ping"
      HealthCheckProtocol: "HTTP"
      HealthyThresholdCount: 2
      Port: 8080
      Protocol: "HTTP"
      UnhealthyThresholdCount: 2
      VpcId:
        "Fn::ImportValue": !Sub "${EnvironmentName}:VPC"
      TargetType: "ip"

  SharedAlbServiceXListenerRule:
    Type: "AWS::ElasticLoadBalancingV2::ListenerRule"
    Properties:
      Actions:
        - Type: "forward"
          TargetGroupArn: !Ref SharedAlbServiceXTargetGroup
      Conditions:
        - Field: "host-header"
          HostHeaderConfig:
            Values:
              - !Ref HttpApiCustomDomain
      ListenerArn: !Ref SharedApplicationLoadBalancerListener
      Priority: !Ref SharedAlbListenerRulePriority

3. VPC link:

  PrivateApiGatewayVpcLinkSecurityGroup:
    Condition: PrivateAccess
    Type: "AWS::EC2::SecurityGroup"
    Properties:
      VpcId:
        'Fn::ImportValue': !Sub "${EnvironmentName}:VPC"
      GroupName: !Sub "${EnvironmentName}-apigw-vpclink"
      GroupDescription: !Sub "SG for API Gateway private VPC link in ${EnvironmentName} environment"
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 80
        ToPort: 80
        CidrIp: 0.0.0.0/0

  PrivateApiGatewayVpcLink:
    Condition: PrivateAccess
    Type: "AWS::ApiGatewayV2::VpcLink"
    Properties:
      Name: !Sub "${EnvironmentName}-api-gateway-vpclink"
      SecurityGroupIds:
        - !Ref PrivateApiGatewayVpcLinkSecurityGroup
      SubnetIds:
        - "Fn::ImportValue": !Sub "${EnvironmentName}:PrivateSubnet1"
        - "Fn::ImportValue": !Sub "${EnvironmentName}:PrivateSubnet2"

4. API Gateway Integration

  HttpApiIntegration:
    Type: "AWS::ApiGatewayV2::Integration"
    Properties:
      ApiId: !Ref HttpApi
      Description: !Sub "Private ALB Integration for ${ServiceName} in ${EnvironmentName} env"
      IntegrationType: "HTTP_PROXY"
      IntegrationMethod: "ANY"
      ConnectionType: "VPC_LINK"
      ConnectionId: !Ref PrivateApiGatewayVpcLink
      IntegrationUri: !Ref SharedApplicationLoadBalancerListener
      PayloadFormatVersion: "1.0"