ASP.NET SignalR with Redis scaleout/backplane causes linear increase in server load

Question

I am running an ASP.NET Web application (.NET Framework 4.7.1) on two different Windows Servers. Each application instance has a SignalR message server with nine different hubs for different types of messages. When both servers are online, there is a SignalR scaleout using the Redis backplane.

Using

• .NET Framework 4.7.1
• ASP.NET SignalR version 2.4.2 (JS and .NET clients also use this version)
• StackExchange.Redis version 2.2.62 (previously used StackExchange.Redis.StrongName 1.2.6)
• Redis server 3.2.12 - cluster support disabled

There are no problems when either of the servers is running by itself (no backplane); however, when both servers are running together, the amount of data in Redis keeps increasing and the amount of memory each server is using keeps growing until some applications stop working or there are noticeable delays in the clients receiving messages. This could take anywhere from one day to one week.

Setup

• Two servers active behind a proxy

• Each server has its app pool recycled once per day at different times

• Few hundred SignalR JS and SignalR.NET clients at any given time, each connected to one or more hubs

• Not using sticky sessions (this is necessary with .NET Core but not with ASP)

• Only web socket protocol is used

• Different channels in Redis are used for other messages between servers like when a client connects or disconnects

• All messages are sent from the server(s) to the clients, one-way

        GlobalHost.Configuration.ConnectionTimeout = TimeSpan.FromSeconds(110);
        GlobalHost.Configuration.DisconnectTimeout = TimeSpan.FromSeconds(30);
        GlobalHost.Configuration.KeepAlive = TimeSpan.FromSeconds(10);
        GlobalHost.Configuration.DefaultMessageBufferSize = 32;  // This is the minimum size
  
        GlobalHost.DependencyResolver.UseStackExchangeRedis(new RedisScaleoutConfiguration(RedisHostName, RedisAppNameKey));
  
        var hubConfiguration = new HubConfiguration { EnableDetailedErrors = true };
  
        app.MapSignalR(hubConfiguration);
  

There are no exceptions or errors in the logs and I cannot reproduce the issue in another environment including with load testing with a significantly higher number of clients and messages.

Looking at SignalR Redis messages, I noticed that some are being duplicated multiple times. Looking at the proxy logs, I noticed there are continuous SignalR abort and negotiate messages even with one server running. When logging client connects and disconnects, there are sometimes duplicate connections for the same ConnectionId for the same Hubs.

Network profile of Redis. Falls are when the app pool recycles.

Memory on the server before it starts having problems. SignalR is taking up most of it.

Edit 1: I took a SignalR trace of both servers for about an hour. bus.log.txt is clean with no exceptions until the app pool was recycled. transports.log.txt has multiple exceptions all through the log on both servers.

SignalR.Transports.WebSocketTransport Error: 0 : OnError(7824eb1d-5916-4295-afe2-632fc8c01130, System.Net.WebSockets.WebSocketException (0x80070026): Reached the end of the file

SignalR.Transports.WebSocketTransport Error: 0 : OnError(d8f299ca-d658-40e8-af83-ba4bbc0166f2, System.Net.WebSockets.WebSocketException (0x800703E3): The I/O operation has been aborted because of either a thread exit or an application request

Update: According to the SignalR development team, the exceptions are normal and are not the cause of my issue.