CODEWITHSUNDEEP
kubernetesnginxwebsockethaproxyjelastic
Unik6065
Unik6065

Reputation: 103

Trouble Connecting WebSocket (ActionCable) in Kubernetes with HAProxy and NGINX Ingress

I am currently developing an application deployed on a Kubernetes cluster hosted in Jelastic Cloud. My topology is as follows:

I'm using an NGINX Ingress Controller, and the API balancers are running HAProxy 2.9.0. The Kubernetes version is v1.28.0.

Problem:

I’m trying to establish a WebSocket connection between my frontend (Next.js) and my backend (Ruby on Rails using Puma 6.4.2). My setup looks like this:

Here's a snippet from my Ingress configuration in Kubernetes:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: swaplit-ingress-websocket
  namespace: swaplit
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /cable$2
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
spec:
  tls:
  - hosts:
    - ws.test.com
    secretName: test-websocket-tls
  ingressClassName: nginx
  rules:
  - host: ws.test.com
    http:
      paths:
      - pathType: Prefix
        path: /cable(/|$)(.*)
        backend:
          service:
            name: test
            port:
              number: 3001

When I check the logs of my backend server, I can see the following message:

Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: upgrade, HTTP_UPGRADE: websocket)

But immediately after, I get this:

Finished "/cable" [WebSocket]

In the Network tab of the browser's dev tools, I notice that the connection status shows Finished, instead of the expected 101 Switching Protocols, so it seems like the WebSocket connection isn’t being persisted between my client and server.

What I suspect:

Due to the Jelastic production environment, my Kubernetes cluster is running behind an HAProxy API balancer. I suspect the issue might be due to a misconfiguration in HAProxy, so I modified the HAProxy configuration to support WebSockets as follows:

#### MAKE CHANGES HERE ONLY IF YOU REALLY KNOW WHAT YOU ARE DOING #####
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global

log 127.0.0.1   local0
user haproxy
group haproxy

ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

ssl-dh-param-file /etc/haproxy/dhparam.pem

pidfile     /var/run/haproxy.pid
tune.ssl.default-dh-param 2048
maxconn     10000
daemon
stats timeout 2m
stats socket /var/run/haproxy.sock mode 660 level admin

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------

defaults

mode                    http
log                     global
option                  httplog
option                  dontlognull
option http-server-close
option                  redispatch
retries                 3
timeout http-request    10s
timeout queue           1m
timeout connect         10s
timeout client          1m
timeout server          "${PROXY_READ_TIMEOUT}"s
timeout http-keep-alive 10s
timeout check           10s
maxconn                 10000

frontend ft_http

#bind :::80 v4v6
mode http
stats enable
stats auth xx:xxxx
stats refresh 30s
stats show-node
stats uri  /haproxy_adm_panel
stats admin if TRUE
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header HTTPS on if { ssl_fc }
http-request set-header Ssl-Offloaded 1 if { ssl_fc }

# Detect WebSocket requests
acl is_websocket hdr(Upgrade) -i websocket
use_backend ws_backend if is_websocket

default_backend default

backend default
mode http
errorfile 503 /etc/haproxy/welcome.http

backend bk_http ###HOSTS ARE ADDED TO THIS BACKEND BY DEFAULT

mode http
cookie SRVNAME insert
balance roundrobin


# Backend for WebSocket
backend ws_backend
mode http
option http-server-close
option forwardfor
balance roundrobin
option http-server-close
option forwardfor

# Set timeouts for WebSockets
timeout connect 5s
timeout client 3600s
timeout server 3600s
timeout tunnel 3600s  # Allow WebSocket tunnel to stay open

## websocket protocol validation
acl hdr_connection_upgrade hdr(Connection)                 -i upgrade
acl hdr_upgrade_websocket  hdr(Upgrade)                    -i websocket
acl hdr_websocket_key      hdr_cnt(Sec-WebSocket-Key)      eq 1
acl hdr_websocket_version  hdr_cnt(Sec-WebSocket-Version)  eq 1
http-request deny if ! hdr_connection_upgrade ! hdr_upgrade_websocket ! hdr_websocket_key ! hdr_websocket_version

## ensure our application protocol name is valid
## (don't forget to update the list each time you publish new applications)
acl ws_valid_protocol hdr(Sec-WebSocket-Protocol) echo-protocol
http-request deny if ! ws_valid_protocol

## websocket health checking
# option httpchk GET / HTTP/1.1rnHost:\ ws.swaplit.org\r\nConnection:\ Upgrade\r\nUpgrade:\ websocket\r\nSec-WebSocket-Key:\ haproxy\r\nSec-WebSocket-Version:\ 13\r\nSec-WebSocket-Protocol:\ echo-protocol
http-check send meth GET uri Host:\ ws.test.com\r\nConnection:\ Upgrade\r\nUpgrade:\ websocket\r\nSec-WebSocket-Key:\ haproxy\r\nSec-WebSocket-Version:\ 13\r\nSec-WebSocket-Protocol:\ echo-protocol ver HTTP/1.0
http-check expect status 101
server wk_1 xx.xxx.xx.xx
server wk_2 xx.xxx.xx.xx
server wk_3 xx.xxx.xx.xx

###TCP SECTION###

frontend tcp_CPlanebalancing_UAQJX
bind *:6443 v4v6
mode tcp
default_backend tcp_CPlanebalancing_UAQJX_backend

backend tcp_CPlanebalancing_UAQJX_backend
mode tcp
balance roundrobin
option tcp-check
server tcp_CPlanebalancing_UAQJX_1 xx.xxx.xx.xxx:xxxx check fall 3 rise 2
server tcp_CPlanebalancing_UAQJX_2 xx.xxx.xx.xxx:xxxx  check fall 3 rise 2
server tcp_CPlanebalancing_UAQJX_3 xx.xxx.xx.xxx:xxxx  check fall 3 rise 2

However, I’m still encountering the same issue. I followed this guide on configuring HAProxy for WebSockets, but I may have made a mistake in my config.

Request: Does anyone have experience setting up WebSockets (especially with ActionCable) on Kubernetes behind an HAProxy balancer? Any tips on how to properly configure the HAProxy or further debug this issue would be greatly appreciated, as I’ve been struggling with this problem for weeks.

Upvotes: 0

Views: 104

Answers (0)

Related Questions