Api Gateway websocket $connect gets a 400 from the http integration point but it works for other routes

Question

I have a problem with my HTTP integration for an AWS API Gateway that uses WEBSOCKET

These are the main characteristics of my configuration (it can be viewed in detail in the cloud formation template at the end of the post):

• Api Gateway Websocket
• Route selection expression $request.body.action
• Disabled execute api endpoint because I'm using a custom domain, although it does not seem to make any difference if I use the Api Gateway direct url instead
• Routes $connect, a send and a $disconnect
• Integration type is HTTP_PROXY
• The integration Uri (and here is the interesting part) is a URL that points to my custom domain, and the DNS resolves as ANOTHER Api Gateway in my AWS account (an HTTP one) that integrates with a private ALB through a VPC_LINK and reaches a web service in an ECS cluster (I guess this is irrelevant now).
• Bot Api Gateway, the http one and the websocket one, use a custom domain api.mycompany.io and ws.mycompany.io with a TLS certificate *.mycompany.io
• The HTTP services run in a private subnet, but they're perfectly reachable from internet. I can send http requests and get back responses.

When I do

wscat -c wss://ws.mycompany.io

I get

error: Unexpected server response: 400

I can see a cloudwatch log without much information

{"requestTime":"21/May/2021:15:20:06 +0000","requestId":"fr1iDG7CjoEFWTg=","httpMethod":"-","path":"-","routeKey":"$connect","status":400,"responseLatency":-}

There is another log group automatically created for that websocket api /aws/apigateway/i4vdq18wg9/production that contains the following

(fr1iDG7CjoEFWTg=) Extended Request Id: fr1iDG7CjoEFWTg=
(fr1iDG7CjoEFWTg=) Verifying Usage Plan for request: fr1iDG7CjoEFWTg=. API Key:  API Stage: i4vdq18wg9/production
(fr1iDG7CjoEFWTg=) API Key  authorized because route '$connect' does not require API Key. Request will not contribute to throttle or quota limits
(fr1iDG7CjoEFWTg=) Usage Plan check succeeded for API Key  and API Stage i4vdq18wg9/production
(fr1iDG7CjoEFWTg=) Starting execution for request: fr1iDG7CjoEFWTg=
(fr1iDG7CjoEFWTg=) WebSocket Request Route: [$connect]
(fr1iDG7CjoEFWTg=) Client [UserAgent: null, SourceIp: 91.194.63.143] is connecting to WebSocket API [i4vdq18wg9].
(fr1iDG7CjoEFWTg=) Endpoint request URI: https://api.mycompany.io/MyCompany.MyService/ws/connect
(fr1iDG7CjoEFWTg=) Endpoint request headers: {Sec-WebSocket-Key=7kp7mwI9YH+WjoGeiGjMEg==, x-amzn-apigateway-api-id=i4vdq18wg9, stage=production, domainName=ws.mycompany.io, X-Forwarded-Proto=https, User-Agent=AmazonAPIGateway_i4vdq18wg9, connectionId=fr1iDchODoECFIQ=, Sec-WebSocket-Version=13, X-Forwarded-For=91.194.63.143, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-60a7cfa6-45d1a7387a071f9c6547dfad, Sec-WebSocket-Extensions=permessage-deflate; client_max_window_bits}
(fr1iDG7CjoEFWTg=) Endpoint request body after transformations: 
(fr1iDG7CjoEFWTg=) Sending request to https://api.mycompany.io/MyCompany.MyService/ws/connect
(fr1iDG7CjoEFWTg=) Received response. Status: 400, Integration latency: 8 ms
(fr1iDG7CjoEFWTg=) Endpoint response headers: {Server=awselb/2.0, Date=Fri, 21 May 2021 15:20:06 GMT, Content-Type=application/json; charset=utf-8, Content-Length=11, Connection=keep-alive}
(fr1iDG7CjoEFWTg=) Endpoint response body before transformations: Bad Request
(fr1iDG7CjoEFWTg=) Client [UserAgent: null, SourceIp: 91.194.63.143] failed to connect to API [i4vdq18wg9]. 

So it seems that the request to the http service, as per the integration, is failing with a 400 Bad Request. No idea why.

I decide to modify the endpoint just for the $connect integration, the send will point to my service as it was before. I change the IntegrationUri to point to beeceptor, so that I can see how the $connect request is arriving at the service: https://mycompanyinterceptor.free.beeceptor.com/ws/connect

And I try to connect a websocket again.

wscat -c wss://ws.mycompany.io

Connected (press CTRL+C to quit)
>

Now it works. So either the my service does not handle $connect properly (can't see why), or the URL has something special as it makes a call to another API Gateway, behind a custom domain..

I can see at beeceptor, that the $connect is arriving as a GET request to the with the following headers

{
  "stage": "production",
  "domainname": "ws.mycompany.io",
  "connectionid": "fr2_RcLUDoECEWQ=",
  "sec-websocket-extensions": "permessage-deflate; client_max_window_bits",
  "sec-websocket-key": "iOkdWsyZw/P8GjSwXwHrbg==",
  "sec-websocket-version": "13",
  "x-amzn-trace-id": "Root=1-60a7d1fb-4c42b6e717aa2c666ecae2ab",
  "x-forwarded-for": "91.194.63.143",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https",
  "user-agent": "AmazonAPIGateway_i4vdq18wg9",
  "x-amzn-apigateway-api-id": "i4vdq18wg9"
}

and Beeceptor is responding with a 200 OK, the following headers

{
  "content-type": "text/plain",
  "access-control-allow-origin": "*",
  "vary": "Accept-Encoding"
}

and this body

Hey ya! Great to see you here. Btw, nothing is configured for this request path. Create a rule and start building a mock API.

Now, I decide to send a message through my websocket that should be forwarded to my send route, and my http integration service

$ wscat -c wss://ws.rubiko.io
Connected (press CTRL+C to quit)
> {'action': 'send', 'data': 'my service should just reply with a 200 OK and the path invoked'}
< OK: /ws/send
>

As you can see, the problem is not with connectivity to my service because this other websocket request arrives properly.

What am I doing wrong? As far as I can tell, the "different" things that could affect $connect are:

• It is translated into a GET request by the API Gateway, without body content. Maybe the http headers are making my service fail for whatever reason?
• I have commented out the $connect integration response. It does not seem to make any difference though.

Running out of ideas here :-(

---

CloudFormation template

AWSTemplateFormatVersion: 2010-09-09

Description: API Gateway for HTTP and Websocket with custom domains

Parameters:
  hostedZoneId:
    Type: AWS::Route53::HostedZone::Id
    Description: the public hosted zone
    Default: Z0561635YJ0IZJCMK5TF
  certificateArn:
    Type: String
    Description: The TLS certificate ARN
    Default: arn:aws:acm:eu-west-1:754027052283:certificate/35b3a270-4dc3-4935-b108-d22dff688b1f
  prefix:
    Type: String
    Description: The prefix namespace or company name
    Default: mycompany

Mappings:
  gateways:
    api:
      domainName: api.mycompany.io
      logGroupName: /aws/api-mycompany/mycompany-http-gateway-logs
    ws:
      domainName: ws.mycompany.io
      logGroupName: /aws/api-mycompany/mycompany-ws-gateway-logs

Resources:
  privateLink:
    Type: AWS::ApiGatewayV2::VpcLink
    Properties:
      Name: !Sub ${prefix}-private-link
      SecurityGroupIds:
        - Fn::ImportValue: !Sub ${prefix}-web-sg-id
      SubnetIds:
        - Fn::ImportValue: !Sub ${prefix}-private-a-id
        - Fn::ImportValue: !Sub ${prefix}-private-b-id
        - Fn::ImportValue: !Sub ${prefix}-private-c-id
  
  apiGatewayAccountConfig:
    Type: "AWS::ApiGateway::Account"
    Properties:
      CloudWatchRoleArn: !GetAtt apiGatewayWatchLogsRole.Arn

  apiGatewayWatchLogsRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ${prefix}-api-gateway-logs
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs

  # Microservices Api Gateway
  apiGatewayDomain:
    Type: AWS::ApiGatewayV2::DomainName
    Properties:
      DomainName: !FindInMap [gateways, api, domainName]
      DomainNameConfigurations:
        - CertificateArn: !Ref certificateArn
          CertificateName: !Sub ${prefix}-certificate
          SecurityPolicy: TLS_1_2

  apiGatewayLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !FindInMap [gateways, api, logGroupName]

  apiGateway:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      Name: !Sub ${prefix}-http-gateway
      Description: Api Gateway for http
      ProtocolType: HTTP
      DisableExecuteApiEndpoint: true

  apiRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref apiGateway
      RouteKey: ANY /{proxy+}
      Target: !Join
        - /
        - - integrations
          - !Ref apiAlbIntegration

  apiAlbIntegration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref apiGateway
      Description: Private ALB integration on 80
      IntegrationType: HTTP_PROXY
      IntegrationMethod: ANY
      ConnectionType: VPC_LINK
      ConnectionId: !Ref privateLink
      IntegrationUri: 
        Fn::ImportValue: !Sub ${prefix}-alb-http-listener-id
      PayloadFormatVersion: 1.0

  apiStage:
    Type: AWS::ApiGatewayV2::Stage
    Properties:
      StageName: $default
      AutoDeploy: true
      ApiId: !Ref apiGateway
      AccessLogSettings:
        DestinationArn: !GetAtt apiGatewayLogGroup.Arn
        Format: '{"requestTime":"$context.requestTime","requestId":"$context.requestId","httpMethod":"$context.httpMethod","path":"$context.path","routeKey":"$context.routeKey","status":$context.status,"responseLatency":$context.responseLatency}'
  
  apiMapping:
    Type: AWS::ApiGatewayV2::ApiMapping
    DependsOn: apiGatewayDomain
    Properties: 
      ApiId: !Ref apiGateway
      DomainName: !FindInMap [gateways, api, domainName]
      Stage: !Ref apiStage
  
  aliasDns:
    Type: AWS::Route53::RecordSet
    Properties:
      Comment: DNS entry for public custom domain to alias microservices API URL
      HostedZoneId: !Ref hostedZoneId
      Name: !FindInMap [gateways, api, domainName]
      Type: A
      AliasTarget:
        DNSName: !GetAtt apiGatewayDomain.RegionalDomainName
        HostedZoneId: !GetAtt apiGatewayDomain.RegionalHostedZoneId
        EvaluateTargetHealth: true
  
  # Websocket Api Gateway
  wsGatewayDomain:
    Type: AWS::ApiGatewayV2::DomainName
    Properties:
      DomainName: !FindInMap [gateways, ws, domainName]
      DomainNameConfigurations:
        - CertificateArn: !Ref certificateArn
          CertificateName: !Sub ${prefix}-certificate
          SecurityPolicy: TLS_1_2
  
  wsApiGatewayLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !FindInMap [gateways, ws, logGroupName]

  wsApiGateway:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      Name: !Sub ${prefix}-ws-gateway
      Description: Api Gateway for Websockets
      ProtocolType: WEBSOCKET
      RouteSelectionExpression: $request.body.action
      DisableExecuteApiEndpoint: true

  wsConnectRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref wsApiGateway
      RouteKey: $connect
      AuthorizationType: NONE
      OperationName: ConnectRoute
      # RouteResponseSelectionExpression: $default
      Target: !Join
        - /
        - - integrations
          - !Ref wsConnectIntegration

  wsConnectIntegration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref wsApiGateway
      Description: Websocket $connect integration
      IntegrationType: HTTP_PROXY
      IntegrationMethod: ANY
      IntegrationUri: https://api.mycompany.io/MyCompany.MyService/ws/connect
      RequestParameters:
        "integration.request.header.domainName": "context.domainName"
        "integration.request.header.stage": "context.stage"
        "integration.request.header.connectionId": "context.connectionId"
      PayloadFormatVersion: 1.0

  # wsConnectRouteResponse:
  #   Type: AWS::ApiGatewayV2::RouteResponse
  #   Properties:
  #     ApiId: !Ref wsApiGateway
  #     RouteId: !Ref wsConnectRoute
  #     RouteResponseKey: $default

  wsSendRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref wsApiGateway
      RouteKey: send
      AuthorizationType: NONE
      OperationName: SendRoute
      RouteResponseSelectionExpression: $default
      Target: !Join
        - /
        - - integrations
          - !Ref wsSendIntegration

  wsSendIntegration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref wsApiGateway
      Description: Websocket send integration
      IntegrationType: HTTP_PROXY
      IntegrationMethod: ANY
      IntegrationUri: https://api.mycompany.io/MyCompany.MyService/ws/send
      RequestParameters:
        "integration.request.header.domainName": "context.domainName"
        "integration.request.header.stage": "context.stage"
        "integration.request.header.connectionId": "context.connectionId"
      PayloadFormatVersion: 1.0

  wsSendRouteResponse:
    Type: AWS::ApiGatewayV2::RouteResponse
    Properties:
      ApiId: !Ref wsApiGateway
      RouteId: !Ref wsSendRoute
      RouteResponseKey: $default

  wsDisconnectRoute:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref wsApiGateway
      RouteKey: $disconnect
      AuthorizationType: NONE
      OperationName: DisconnectRoute
      RouteResponseSelectionExpression: $default
      Target: !Join
        - /
        - - integrations
          - !Ref wsDisconnectIntegration

  wsDisconnectIntegration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref wsApiGateway
      Description: Websocket $disconnect integration
      IntegrationType: HTTP_PROXY
      IntegrationMethod: ANY
      IntegrationUri: https://api.mycompany.io/MyCompany.MyService/ws/disconnect
      RequestParameters:
        "integration.request.header.domainName": "context.domainName"
        "integration.request.header.stage": "context.stage"
        "integration.request.header.connectionId": "context.connectionId"
      PayloadFormatVersion: 1.0

  wsDisconnectRouteResponse:
    Type: AWS::ApiGatewayV2::RouteResponse
    Properties:
      ApiId: !Ref wsApiGateway
      RouteId: !Ref wsDisconnectRoute
      RouteResponseKey: $default

  wsApiStage:
    Type: AWS::ApiGatewayV2::Stage
    DependsOn:
      - wsConnectRoute
      - wsSendRoute
      - wsDisconnectRoute
    Properties:
      StageName: production
      Description: Autodeploy in production
      AutoDeploy: true
      ApiId: !Ref wsApiGateway
      AccessLogSettings:
        DestinationArn: !GetAtt wsApiGatewayLogGroup.Arn
        Format: '{"requestTime":"$context.requestTime","requestId":"$context.requestId","httpMethod":"$context.httpMethod","path":"$context.path","routeKey":"$context.routeKey","status":$context.status,"responseLatency":$context.responseLatency}'
  
  wsApiMapping:
    Type: AWS::ApiGatewayV2::ApiMapping
    DependsOn: wsGatewayDomain
    Properties: 
      ApiId: !Ref wsApiGateway
      DomainName: !FindInMap [gateways, ws, domainName]
      Stage: !Ref wsApiStage
  
  wsAliasDns:
    Type: AWS::Route53::RecordSet
    Properties:
      Comment: DNS entry for public custom domain to alias Websocket URL
      HostedZoneId: !Ref hostedZoneId
      Name: !FindInMap [gateways, ws, domainName]
      Type: A
      AliasTarget:
        DNSName: !GetAtt wsGatewayDomain.RegionalDomainName
        HostedZoneId: !GetAtt wsGatewayDomain.RegionalHostedZoneId
        EvaluateTargetHealth: true

Outputs:
  apiUri:
    Description: Http Api URL
    Value: 
      !Sub 
        - https://${domainName}
        - domainName: !FindInMap [gateways, api, domainName]
  wsUri:
    Description: Websocket Api URL
    Value:
      !Sub 
        - wss://${domainName}/web/index.html
        - domainName: !FindInMap [gateways, ws, domainName]

Just in case, my service, at the moment is a C# ASP.NET app, with middleware that does only this

public async Task Invoke(HttpContext context)
{
  var request = context.Request;
  if (request.Path.ToString().Contains("/ws/", StringComparison.InvariantCultureIgnoreCase))
  {
    var path = request.Path.ToString();
    _logger.LogDebug("Received websocket message on {path}");
    await context.Response.WriteAsync($"OK: {path}");
  }
  else
  {
    // Call the next delegate/middleware in the pipeline
    await _next(context);
    var response = context.Response;
  }

---

UPDATE 1: I have tried without middleware in my service. Just a plain old GET endpoint https://api.mycompany.io/MyCompany.MyService/api/sample that returns 200 OK when invoked. It's up and running, and from POSTman I can verify it works fine.

However it still fails to connect the websocket. These are the traces.

(f0sygHFWDoEFuyQ=) Extended Request Id: f0sygHFWDoEFuyQ=
(f0sygHFWDoEFuyQ=) Verifying Usage Plan for request: f0sygHFWDoEFuyQ=. API Key:  API Stage: rui1tsyre5/production
(f0sygHFWDoEFuyQ=) API Key  authorized because route '$connect' does not require API Key. Request will not contribute to throttle or quota limits
(f0sygHFWDoEFuyQ=) Usage Plan check succeeded for API Key  and API Stage rui1tsyre5/production
(f0sygHFWDoEFuyQ=) Starting execution for request: f0sygHFWDoEFuyQ=
(f0sygHFWDoEFuyQ=) WebSocket Request Route: [$connect]
(f0sygHFWDoEFuyQ=) Client [UserAgent: null, SourceIp: 91.194.63.143] is connecting to WebSocket API [rui1tsyre5].
(f0sygHFWDoEFuyQ=) Endpoint request URI: https://api.mycompany.io/MyCompany.MyService/api/sample
(f0sygHFWDoEFuyQ=) Endpoint request headers: {Sec-WebSocket-Key=rzw3hU/DdJC5x7ujeuRgCQ==, x-amzn-apigateway-api-id=rui1tsyre5, stage=production, domainName=ws.mycompany.io, X-Forwarded-Proto=https, User-Agent=AmazonAPIGateway_rui1tsyre5, connectionId=f0sygdPvDoECHRg=, Sec-WebSocket-Version=13, X-Forwarded-For=91.194.63.143, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-60ab5b43-5653a1fe637f11634b1e16ce, Sec-WebSocket-Extensions=permessage-deflate; client_max_window_bits}
(f0sygHFWDoEFuyQ=) Endpoint request body after transformations: 
(f0sygHFWDoEFuyQ=) Sending request to https://api.mycompany.io/MyCompany.MyService/api/sample
(f0sygHFWDoEFuyQ=) Received response. Status: 400, Integration latency: 4 ms
(f0sygHFWDoEFuyQ=) Endpoint response headers: {Server=awselb/2.0, Date=Mon, 24 May 2021 07:52:35 GMT, Content-Type=application/json; charset=utf-8, Content-Length=11, Connection=keep-alive}
(f0sygHFWDoEFuyQ=) Endpoint response body before transformations: Bad Request
(f0sygHFWDoEFuyQ=) Client [UserAgent: null, SourceIp: 91.194.63.143] failed to connect to API [rui1tsyre5]. 

Why is the integration returning a 400 error? It'd be great if I could see more details on the error.

Answer 1

I solved it.

The reason it failed with the "real" http integration pointing to my service is due to some http headers.

I reproduced the issue by accessing the cloud watch logs and viewing all the http headers that were being used in the actual http integration call, and, in PostMan, making a request to my service with exactly those headers. As expected, I got 400 Bad Request.

(fr1iDG7CjoEFWTg=) Endpoint request headers: {Sec-WebSocket-Key=7kp7mwI9YH+WjoGeiGjMEg==, x-amzn-apigateway-api-id=i4vdq18wg9, stage=production, domainName=ws.mycompany.io, X-Forwarded-Proto=https, User-Agent=AmazonAPIGateway_i4vdq18wg9, connectionId=fr1iDchODoECFIQ=, Sec-WebSocket-Version=13, X-Forwarded-For=91.194.63.143, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-60a7cfa6-45d1a7387a071f9c6547dfad, Sec-WebSocket-Extensions=permessage-deflate; client_max_window_bits}

First of all, there is no Host http header, not sure why. Also, the Sec-WebSocket-Version=13 seems to cause a 400 error even when there's a host header.

There are several options to fix this. At the service I can manipulate the http headers in a middleware to accommodate it, or I can tweak the API Gateway. I did the latter, and set a static value for the host header and set the Sec-WebSocket-Version to 0 (I don't know how to remove the header, so I overwrite it with a dummy value)

In cloudFormation's integration

RequestParameters:
  "integration.request.header.domainName": "context.domainName"
  "integration.request.header.stage": "context.stage"
  "integration.request.header.connectionId": "context.connectionId"
  "integration.request.header.host": "'api.mycompany.io'"
  "integration.request.header.Sec-WebSocket-Version": "'0'"