CODEWITHSUNDEEP
redisstackexchange.redis
infojolt
infojolt

Reputation: 5418

Stack Exchange Redis client timeout

I'm struggling to work out what this error message from the Stack Exchange Redis client is telling me:

Unhandled Exception: System.AggregateException: One or more errors occurred. (Timeout awaiting response (outbound=0KiB, inbound=0KiB, 32516ms elapsed, timeout is 30000ms), inst: 0, qu: 0, qs: 1, in: 0, serverEndpoint: Unspecified/XXX:6379, mgr: 9 of 10 available, clientName: YYY, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)) ---> StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 32516ms elapsed, timeout is 30000ms), inst: 0, qu: 0, qs: 1, in: 0, serverEndpoint: Unspecified/XXX:6379, mgr: 9 of 10 available, clientName: YYY, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts

I'm using this code to select the least loaded connection multiplexer and back off, but I'm still getting timeouts when firing a large number of string set commands.

public class RedisConnectionManager : IRedisConnectionManager
{
    private const int MaxQueueLength = 10;
    private readonly List<Lazy<ConnectionMultiplexer>> _connectionMultiplexers;

    public RedisConnectionManager(List<Lazy<ConnectionMultiplexer>> connectionMultiplexers)
    {
        this._connectionMultiplexers = connectionMultiplexers;
    }

    public async Task<ConnectionMultiplexer> GetLeastBusyConnectionAsync()
    {
        var leastBusyConnection = this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First();
        await WaitUntilConnectionAvailableAsync(leastBusyConnection);
        return this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First().Value;
    }

    private static async Task WaitUntilConnectionAvailableAsync(Lazy<ConnectionMultiplexer> leastBusyConnection)
    {
        while (leastBusyConnection.Value.GetCounters().Interactive.TotalOutstanding > MaxQueueLength)
        {
            await Task.Delay(100);
        }
    }
}

Upvotes: 2

Views: 2559

Answers (1)

vishal shah
vishal shah

Reputation: 217

There are several answers on SO over regarding this.

In your particular case, it looks like the issue with ThreadPool throtlling, where busy threads are 1025 which are more than Min configured 1024 threads.

WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767)

Have a look here, however beware of the other server performance consequences (depending on your use case) as mentioned in the article. https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md

Upvotes: 1

Related Questions