CODEWITHSUNDEEP
ruby-on-railsnginxwebsocketnginx-ingressactioncable
shoffing
shoffing

Reputation: 117

Rails - ActionCable sporadically failing to subscribe

I'm working on adding websocket support to a large existing Rails app. I'm starting with a simple test just to make sure it can handle our production load - we will need to support many thousands of active connections for something like a notification or chat channel, one for each active user on the site.

I've deployed code that simply creates a subscription to a new "PerfTestChannel" on page load:

App.cable.subscriptions.create('PerfTestChannel', ...)

Which just streams for the current user:

class PerfTestChannel < ApplicationCable::Channel
  def subscribed
    stream_for current_user
  end
end

At low traffic, it seems to work fine. It creates the subscription and confirms it:

enter image description here

At higher traffic, however, it will sporadically fail to subscribe to the channel, and spam subscribe retries twice per second indefinitely:

enter image description here

I cannot figure out why this is failing, and how to fix it. It's strange because, as shown in the picture, ping messages are still transmitting successfully, so the cable connection is clearly open. It just won't subscribe to the PerfTestChannel.

We are using a standard Redis adapter setup. At this point in the test, we have ~200 open connections:

irb(main):016:0> Redis.new.pubsub("channels", "action_cable/*").count
=> 211

We are seeing some errors in the production logs related to web sockets:

enter image description here

The first error seems to be not relevant, because (as mentioned above) it's clear the websocket upgrade was successful because pings are going through fine.

The second error seems like it might be relevant, though. It's coming from this point in the source: https://github.com/rails/rails/blob/main/actioncable/lib/action_cable/connection/base.rb#L94

I don't see how the websocket would be closed, though, because of the ping thing. So this might be a red herring error as well?

We are using load balancing through nginx-ingress, which claims to not need any special configuration for websockets beyond upping the timeouts to ~1 hour, which we have already done, to no avail.

Upvotes: 2

Views: 171

Answers (1)

Ronak Bhatt
Ronak Bhatt

Reputation: 113

I guess you can try following two changes

  1. Increase the number of Redis connections. You can do this by setting the pool_size option in your config/cable.yml file, like so:
production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/1" } %>
  pool_size: 50
  1. Check your nginx-ingress configuration. Check that the proxy_read_timeout and proxy_send_timeout values are set to a high enough value (e.g. 1 hour) and that the proxy_http_version value is set to 1.1.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
spec:
  rules:
    - host: mydomain.com
      http:
        paths:
          - path: /websocket
            backend:
              serviceName: my-service
              servicePort: 80

Upvotes: 0

Related Questions