CODEWITHSUNDEEP
amazon-web-servicesasp.net-core-1.0stackexchange.redisamazon-elasticachetimeoutexception
mukesh joshi
mukesh joshi

Reputation: 604

stackexchange.redis with .net core 1.1 timeout

I have .net core 1.1 based web application and using Amazon ElastiCache of Redis for cache management. I am using "StackExchange.Redis": "1.1.608" version for cache related work. Following is the code used to establish connection public class CacheManager { private static Lazy connection; private static object lockObject = new Object(); private readonly ILogger logger;

    public CacheManager(IConfiguration config, ILoggerFactory loggerFactory)
    {
        this.logger = loggerFactory.CreateLogger<CacheManager>();
        if (connection == null)
        {
            lock (lockObject)
            {
                if (connection == null)
                {
                   connection = new Lazy<ConnectionMultiplexer>(() =>
                    {
                        return ConnectionMultiplexer.Connect(config["Cache:ConnectionString"]);
                    });
                }
            }
        }
    }

    public static ConnectionMultiplexer Connection
    {
        get
        {
            try
            {
                return connection.Value;
            }
            catch (Exception e)
            {
                return null;
            }
        }
    }}

The cache get method try to fetch using the key and if data not found it just fetches it from database and the set it.

We are getting frequent timeout error on redis on production, and not only that, it's causing client to become unresponsive, i.e., the website gives an IIS 500 error. There is nothing in log and redis logs are very simple:

Failed to get cahced value: Timeout performing GET , inst: 61, queue: 2, qu: 0, qs: 2, qc: 0, wr: 0, wq: 0, in: 207, ar: 0, clientName: (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

Redis is hosted on a separate server instance. How can I get a better idea what is the root cause of this ?

Upvotes: 1

Views: 961

Answers (1)

MichaelBolton
MichaelBolton

Reputation: 84

Mukesh, If the redis server's resources (CPU/RAM/hard drive space) aren't constraining things, I would look at the possibility of the threads allocated to your software locking up (where the errors on the logs are actually just a secondary symptom).

I would make sure your redis factory is a singleton instance registered with the services collection on startup. The "connection" field should be set up like so: 

private static readonly Lazy<ConnectionMultiplexer> connection = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(config["Cache:ConnectionString"]));

This way you never would have to assign it, and it's shared across your application (like intended) with no risk of overrunning your servers with open connections. Your "ConfigureServices" method would have something like the following:

services.AddSingleton<ICacheManager, CacheManager>();

If for some reason you have to lock, instead of locking on an object, await locking on a semaphore slim (figure 10 in this link): https://msdn.microsoft.com/en-us/magazine/jj991977.aspx

If those code changes don't make things better, I would double-check for server resource fluctuations (I don't know what kind of user-base you have, but I've seen website bandwidth become an issue at times). If it turns out it's bandwidth then I would try pub/sub and/or client-side caching.

You could also double check the RAM settings of your redis install. If you override the defaults and then later change the available resources you can run into some problems.

Upvotes: 0

Related Questions