Issue Connecting Next.js Frontend to Ruby on Rails Backend via WebSockets on Kubernetes

Question

I'm working on connecting my Next.js frontend to a Ruby on Rails backend using WebSockets. Both apps are deployed on Kubernetes.

My frontend is accessible at test.com. I have my WebSocket server running at ws.test.com, and I'm redirecting WebSocket requests from ws.test.com to my Rails app on the /cable path. Here’s my Ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress-websocket
  namespace: test
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: /cable$2
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
spec:
  tls:
  - hosts:
    - ws.test.com
    secretName: test-websocket-tls
  ingressClassName: nginx
  rules:
  - host: ws.test.com
    http:
      paths:
      - pathType: Prefix
        path: /cable(/|$)(.*)
        backend:
          service:
            name: test
            port:
              number: 3001

The issue: When I try to connect to the WebSocket, it fails at first, but then retries and succeeds. However, the WebSocket connection still doesn’t function as expected, and the frontend doesn't behave correctly with real-time updates.

Has anyone experienced something similar or have any insights into what might be going wrong?

Logs:

[...]

I, [2024-09-19T08:45:24.541417 #40]  INFO -- : [1b31e70ed2abac80ee01202824173028] Started GET "/cable" for xx.xxx.xxx.x at 2024-09-19 08:45:24 +0000
I, [2024-09-19T08:45:24.541806 #40]  INFO -- : [1b31e70ed2abac80ee01202824173028] Started GET "/cable" [WebSocket] for xx.xxx.xxx.x at 2024-09-19 08:45:24 +0000
E, [2024-09-19T08:45:24.541877 #40] ERROR -- : [1b31e70ed2abac80ee01202824173028] Request origin not allowed: 
E, [2024-09-19T08:45:24.541911 #40] ERROR -- : [1b31e70ed2abac80ee01202824173028] Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: upgrade, HTTP_UPGRADE: websocket)
I, [2024-09-19T08:45:24.541947 #40]  INFO -- : [1b31e70ed2abac80ee01202824173028] Finished "/cable" [WebSocket] for xx.xxx.xxx.x at 2024-09-19 08:45:24 +0000
I, [2024-09-19T08:45:24.874860 #21]  INFO -- : [924a5eef0a477036e4441678d384a4db] Started GET "/cable" for x.xxx.xxx.x at 2024-09-19 08:45:24 +0000
I, [2024-09-19T08:45:24.875276 #21]  INFO -- : [924a5eef0a477036e4441678d384a4db] Started GET "/cable" [WebSocket] for x.xxx.xxx.x at 2024-09-19 08:45:24 +0000
I, [2024-09-19T08:45:24.875339 #21]  INFO -- : [924a5eef0a477036e4441678d384a4db] Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: upgrade, HTTP_UPGRADE: websocket)
I, [2024-09-19T08:45:24.877974 #21]  INFO -- : Registered connection (Z2lkOi8vc3dhcGxpdC9Vc2VyLzE)
I, [2024-09-19T08:45:24.915214 #21]  INFO -- : Finished "/cable" [WebSocket] for x.xx.xxx.x at 2024-09-19 08:45:24 +0000

[...]

As you can see, what's strange is that there are two connection attempts: the first one is refused, and the second one is accepted but still closes before any activity occurs.

Is there any configuration I should add to the Rails backend since it's running in a container?

Thank you in advance!

PS: I’m open to any suggestions for better ways to configure the WebSocket connection or troubleshooting tips. I have seen an error saying that the origin request is not allowed, but it's odd because one connection attempt does succeed.

EDIT:

I am using a Kubernetes (k8s) cluster on Jelastic Cloud, which places my cluster behind an HAProxy Load Balancer. I have updated the HAProxy configuration to support WebSocket connections, as shown below:

#### MAKE CHANGES HERE ONLY IF YOU REALLY KNOW WHAT YOU ARE DOING #####
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global

log 127.0.0.1   local0
user haproxy
group haproxy

ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

ssl-dh-param-file /etc/haproxy/dhparam.pem

pidfile     /var/run/haproxy.pid
tune.ssl.default-dh-param 2048
maxconn     10000
daemon
stats timeout 2m
stats socket /var/run/haproxy.sock mode 660 level admin

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------

defaults

mode                    http
log                     global
option                  httplog
option                  dontlognull
option http-server-close
option                  redispatch
retries                 3
timeout http-request    10s
timeout queue           1m
timeout connect         10s
timeout client          1m
timeout server          "${PROXY_READ_TIMEOUT}"s
timeout http-keep-alive 10s
timeout check           10s
maxconn                 10000

frontend ft_http

#bind :::80 v4v6
mode http
stats enable
stats auth xx:xxxx
stats refresh 30s
stats show-node
stats uri  /haproxy_adm_panel
stats admin if TRUE
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header HTTPS on if { ssl_fc }
http-request set-header Ssl-Offloaded 1 if { ssl_fc }

# Detect WebSocket requests
acl is_websocket hdr(Upgrade) -i websocket
use_backend ws_backend if is_websocket

default_backend default

backend default
mode http
errorfile 503 /etc/haproxy/welcome.http

backend bk_http ###HOSTS ARE ADDED TO THIS BACKEND BY DEFAULT

mode http
cookie SRVNAME insert
balance roundrobin


# Backend for WebSocket
backend ws_backend
mode http
option http-server-close
option forwardfor
balance roundrobin
option http-server-close
option forwardfor

# Set timeouts for WebSockets
timeout connect 5s
timeout client 3600s
timeout server 3600s
timeout tunnel 3600s  # Allow WebSocket tunnel to stay open

###TCP SECTION###

frontend tcp_CPlanebalancing_UAQJX
bind *:6443 v4v6
mode tcp
default_backend tcp_CPlanebalancing_UAQJX_backend

backend tcp_CPlanebalancing_UAQJX_backend
mode tcp
balance roundrobin
option tcp-check
server tcp_CPlanebalancing_UAQJX_1 xx.xxx.xx.xxx:xxxx check fall 3 rise 2
server tcp_CPlanebalancing_UAQJX_2 xx.xxx.xx.xxx:xxxx check fall 3 rise 2
server tcp_CPlanebalancing_UAQJX_3 xx.xxx.xx.xxx:xxxx check fall 3 rise 2

However, I suspect the issue might be related to a lack of session persistence. When I check the network tab in my web browser, the WebSocket connection shows as "finished" instead of returning a 101 Switching Protocols status.

For reference, here are the configuration files in my Rails app:

routes.rb

[...]
mount ActionCable.server => "/cable"
[...]

production.rb

[...]
config.action_cable.mount_path = '/cable'
config.action_cable.url = 'wss://ws.example.com/cable'
config.action_cable.allowed_request_origins = ['https://example.com', 'http://example.com', %r{https://example.com.* http://example.com.*}]
[...]

application.rb

[...]
config.action_cable.mount_path = '/cable'
[...]

Last but not least, here are the logs from my frontend pod, which is making the WebSocket request:

Error: Unexpected server response: 404
    at ClientRequest.<anonymous> (/app/node_modules/next/dist/compiled/ws/index.js:1:40987)
    at ClientRequest.emit (node:events:517:28)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:700:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)
    at TLSSocket.socketOnData (node:_http_client:541:22)
    at TLSSocket.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at Readable.push (node:internal/streams/readable:278:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)