CODEWITHSUNDEEP
redisstackexchange.redis
Gil Stal
Gil Stal

Reputation: 3458

redis throws timeout errors

My server started throwing a couple of hours ago the following exception for an unknown reason. It's a web application. Can anyone help please? What can I do?

, Timeout performing GET SomeKey, inst: 1, mgr: ProcessReadQueue, err: never, queue: 12, qu: 0, qs: 12, qc: 0, wr: 0, wq: 0, in: 1702, ar: 1, clientName: SSD41ACCU10147, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=11,Free=8180,Min=4,Max=8191), Local-CPU: unavailable (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

, Timeout performing GET SomeKey2, inst: 2, mgr: ProcessReadQueue, err: never, queue: 33, qu: 0, qs: 33, qc: 0, wr: 0, wq: 0, in: 1141, ar: 1, clientName: SSD41ACCU10147, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=18,Free=8173,Min=4,Max=8191), Local-CPU: unavailable (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

Stacktrace

at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:line 2044 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 81 at StackExchange.Redis.RedisDatabase.HashGetAll(RedisKey key, CommandFlags flags) in C:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 130 at ...

Edit: Is it possible that I have to much connections that are not closing?

I have a static Property:

public static ConnectionMultiplexer Redis = ConnectionMultiplexer.Connect("localhost:1234,abortConnect=false");

And then from each place I need to query the redis I do:

IDatabase db = GlobalStaticClass.redis.GetDatabase();

Is there a problem with this approach?

Edit 2 [Answers to marc]:

  1. I have Some keys that are very large (1MB) and some that are merely numbers or small string. But this is not new.

It's been like that for couple of months. Suddenly today, 2 hours ago, ALL redis requests started throwing exceptions. (The attached exception is just an example)

  1. Not to much. The redis was never close to be the bottle neck.

  2. Slowlog shows nothing intersting.

  3. How can I check that?
  4. Trying that right now.

Edit 3: Reset IIS solves the problem for 15-30 minutes and then it's starts again and never stops.

Edit 4: It was the hosting company fault. Not really related to redis.

It caused all of our servers to be very slow and redis was the first one to return timeout errors.

Your answers did help me to find who is responsible.

Thanks all.

Upvotes: 0

Views: 1879

Answers (2)

Carl Dacosta
Carl Dacosta

Reputation: 891

From your Timeout error message (note that the value for "Busy" is greater than the value of "Min"), you are experiencing Threadpool growth throttling.

Timeout performing GET SomeKey, inst: 1, mgr: ProcessReadQueue, err: never, queue: 12, qu: 0, qs: 12, qc: 0, wr: 0, wq: 0, in: 1702, ar: 1, clientName: SSD41ACCU10147, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=11,Free=8180,Min=4,Max=8191),

Here is an explanation and recommendations on a few things you can try out. https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md

Upvotes: 0

Marc Gravell
Marc Gravell

Reputation: 1062492

The overall approach should be fine; the outbound is clear, we have a small amount of data on the inbound, and we have an active reader - which seems healthy. So we get into context:

  • what is the size of the data in SomeKey?
  • how busy is the server?
  • is slowlog indicating any particularly long-running operations that would explain blocking at the server?
  • is the server log showing long database persistence operations or slaving operations that might explain things?
  • as a last resort: have you tried increasing the sync timeout? (see search for syncTimeout here)

Upvotes: 2

Related Questions