Web Socket wss:// in AWS EKS with nginx Ingress. returns 400. ws://, http:// and https:// works perfectly

I've been trying to run few services in AWS EKS Cluster. I followed the ingress-nginx guide to get https with AWS ACM certificate

https://kubernetes.github.io/ingress-nginx/deploy/#aws

Used tls termination at ingress controller

I used 3 routes for each services as

adminer.xxxx.com - points to an adminer service

socket.xxxx.com - points to the wss service written in nodejs

service.xxxx.com - points to a program that returns a page which connects to socket url

Without TLS Termination, in http:// everything works fine, ws://socket.xxxx.com/socket.io gets connected and responds well.

When I add TLS, the request goes to wss://socket.xxxx.com/socket.io and the nginx returns 400. I Can't figure out why it happens.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations: 
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header Host $http_host;
    # nginx.ingress.kuberenetes.io/use-regex: "true"
spec:
  rules:
    - host: adminer.xxxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: adminer-svc
              servicePort: 8080
    - host: socket.xxxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nodejs-svc
              servicePort: 2020
    - host: service.xxxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: django-svc
              servicePort: 8000

I Tried with and without these configurations

nginx.ingress.kubernetes.io/configuration-snippet: |
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
          proxy_set_header Host $http_host;

Also I've tried changing the socket.xxxx.com into service.xxxx.com and assigned to be forwarded for /socket.io path

I've also put a url in nodejs with express to test if its working at all, and it responds properly in https://

Only the wss:// has the issue.

PS : This entire Service works when nginx is setup in a normal system with nginx configuration

location / {
      proxy_pass http://localhost:2020/;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
}

I tried request like this as well

https://node-socket.xxxx.com/socket.io/?EIO=3&transport=polling this works

https://node-socket.xxxx.comsocket.io/?EIO=3&transport=websocket this doesnt.

Combinations I tried

protocol, balancer, backendproto, transport => result
wss://, ELB, TCP, websocket  =>  400
wss://, NLB, TCP, websocket  =>  400
wss://, ELB, HTTP, websocket  =>  400
wss://, NLB, HTTP, websocket  =>  400
ws://, ELB, TCP, websocket => 400
ws://, ELB, HTTP, websocket => 400
ws://, NLB, TCP, websocket => 400
ws://, NLB, HTTP, websocket => 400

polling worked in every cases

Upvotes: 12