CODEWITHSUNDEEP
multithreadingazureasp.net-corecachingredis
Josh L
Josh L

Reputation: 1492

StackExchange Redis Timeouts and how to set my Minimum Thread Pool Number in .NET Core?

I am getting the following error when trying to query a very simple string (my application log level) from my azure redis cache that just suddenly appeared in my production environment (this query occurs frequently):

Failed to deserialize loglevel from Redis Cache: Timeout performing GET (5000ms), next: GET LoggingLevel:ApplicationAPI, inst: 2, qu: 0, qs: 1289, aw: False, rs: ReadAsync, ws: Idle, in: 130355, serverEndpoint: XXXX mgr: 10 of 10 available, clientName: application-api-67cdbb4cfd-lkqdk, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3515,Free=29252,Min=1,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

I have read the thread that they are suggesting and I have also have read How to troubleshoot Azure Cache for Redis

Based on the metrics I have read in Azure dashboard, I do not believe I am exceeding my current memory or connection limits. It seems to have to do with my number of worker threads set in the thread pool. I am reading that I should:

"set the minimum configuration value for IOCP and WORKER threads to something larger than the default value"

I have also read that I can set my min thread count using ThreadPool.SetMinThreads(Int32, Int32) api in .NET Core.

So basically my questions are:

  1. Is it safe to set my min thread pool when running my application in an AKS cluster?
  2. Where should I set this value? In Program.cs? In Startup.cs?
  3. Based on the log I saw above would it be safe to set my min level to 4000 ? So I am above the Busy threshold for Worker threads?

Upvotes: 13

Views: 12338

Answers (1)

MÇT
MÇT

Reputation: 1143

I will directly try to answer your questions since you have already understand the underlying problem.

  1. It is safe if you know what you are doing (and I think you know what you are doing). If you would run it on another Kubernetes cluster service or even in IIS, it would also be safe.

  2. You can set the value in the Main method of Program.cs.

  3. Yes, you should set it to something above 3515 (busy count in your exception message) and monitor your application and node metrics. You can start with 4000. However, it is a high number (I'm not saying "too high", just "high"). Therefore, you may need to create more pods instead of increasing minimum worker threads value this much. Unfortunately, there is no silver bullet here. You have to try a value, monitor your application (response times, CPU, RAM, etc.), and tweak the value.

Upvotes: 7

Related Questions