How to avoid cache misses with memory cache in .net-core-2.1

Question

How is it possible to update the cache in the background to avoid cache misses?

In .net-core-2.1 I can add a memory cache like so:

public class Startup
{    
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMemoryCache();
    }
}

Then it's very straightforward to use:

[Route("api")]
public class DataController : Controller
{
    private readonly IMemoryCache _cache;
    private readonly DataContext _dataContext;

    public DataController(IMemoryCache cache, DataContext dataContext)
    {
        _cache = cache;
        _dataContext = dataContext;
    }

    [HttpGet]
    [Route("GimmeCachedData")]
    public async Task<IActionResult> Get()
    {
        var cacheEntry = await
            _cache.GetOrCreateAsync("MyCacheKey", entry =>
            {
                entry.AbsoluteExpiration = DateTime.Now.AddSeconds(20);
                return Task.FromResult(_dataContext.GetOrders(DateTime.Now));
            });

        return Ok(cacheEntry);
    }
}

However, after 20 seconds of amazingly fast cached powered bliss infused requests, as expected, the cached item is expired and the next request is stalled because of a cache-miss, and subsequent data loading.

Argh! so the cache only works sometimes. Why not have the option of having it work all the time?

How can I add functionality to:

1. return the old item (in the meantime) AND
2. automatically update the cache when items expire or are noticed to be expired so the next request will get the updated value?

In trying to solve this problem I have encountered 2 main obstacles with my implementation using an IHostedService:

1. When the cached item is expired it's evicted and no longer available; meaning I can't return it.
2. Updating cached items that require the database cause those calls to happen out of scope.

This cache update can kick off either directly after noticing a cache miss, or by actively monitoring for the next item to expire.

I've tried rolling my own cache (adding it as a singleton) using a ConcurrentDictionary<String, CacheItem>. The CacheItem class contains properties for the Value, Expiration, and a Factory (i.e.: a value-returning-delegate). But I found, as this delegate is probably set at request time and called in the IHostedService background thread, it caused context out of scope exception.

Answer 1

I have found a solution that seems to work.

1. Implement an IHostedService (extended from BackgroundService class). This class will serve as the background thread managed by the .net core framework. The background thread will keep the cache updates going (by calling ICache.UpdateCache as explained below), to avoid request-time cache misses.

public class CacheUpdateService : BackgroundService
{
    private readonly ILogger<CacheUpdateService> _logger;
    private readonly IServiceProvider _serviceProvider;
    private readonly ICache _cache;

    public CacheUpdateService(ILogger<CacheUpdateService> logger, IServiceProvider serviceProvider, ICache cache)
    {
        _logger = logger;
        _serviceProvider = serviceProvider;
        _cache = cache;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogDebug("CacheUpdateService is starting.");

        stoppingToken.Register(Dispose);

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                using (var scope = _serviceProvider.CreateScope())
                {
                    var dataContext = scope.ServiceProvider.GetRequiredService<DataContext>();

                    // This tight loop calls the UpdateCache, which will block if no updates are necessary
                    await Task.Run(() => _cache.UpdateCache(dataContext), stoppingToken);
                }
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Exception in the CacheUpdateService");
            }
        }

        _logger.LogDebug("CacheUpdateService has stopped.");
    }

    public override void Dispose()
    {
        using(var scope = _serviceProvider.CreateScope())
        {
            var scopedProcessingService = scope.ServiceProvider.GetRequiredService<ICache>();

            // Dispose here on ICache will release any blocks
            scopedProcessingService.Dispose();
        }

        base.Dispose();
    }
}

2. The Cache class below implements the background UpdateCache method which will update 1 expired item at a time. Prioritizing the one most expired. It also implements request-scoped GetOrCreate method. Note I'm using a delegate (Func<IDataContext, Object>) in the CacheEntry as the value population factory. This allows Cache class to inject a properly scoped DataContext (received from the IHostedService) and it also allows the caller to specify which method of the DataContext is called to get the results of the specific cache key value. Notice I'm using an AutoResetEvent to wait for 1st-time data population as well as a timer to kick off the next cache refresh. This implementation will suffer a cache-miss for the 1st time the item is called (and I guess after it hasn't been called for more than 1 hour; as it will be evicted after 1 hr.).

public class CacheEntry
{
    public String Key { get; set; }
    public Object Value { get; set; }
    public Boolean Updating { get; set; }
    public Int32 ExpirySeconds { get; set; }
    public DateTime Expiration { get; set; }
    public DateTime LastAccessed { get; set; }
    public Func<IDataContext, Object> ValueFactory { get; set; }
}

public interface ICache : IDisposable
{
    void UpdateCache(IDataContext dataContext);
    T GetOrCreate<T>(String key, Func<IDataContext, T> factory, Int32 expirySeconds = 0) where T : class;
}

public class Cache : ICache
{
    private readonly ILogger _logger;
    private readonly ConcurrentDictionary<String, CacheEntry> _cache;
    private readonly AutoResetEvent _governor; 

    public Cache(ILogger<Cache> logger)
    {
        _logger = logger;
        _cache = new ConcurrentDictionary<String, CacheEntry>();
        _governor = new AutoResetEvent(false);
    }

    public void Dispose()
    {
        _governor.Set();
    }

    public static Int32 CacheForHour => 3600;
    public static Int32 CacheForDay => 86400;
    public static Int32 CacheIndefinitely => 0;

    public void UpdateCache(IDataContext dataContext)
    {
        var evictees = _cache.Values
            .Where(entry => entry.LastAccessed.AddHours(1) < DateTime.Now)
            .Select(entry => entry.Key)
            .ToList();
        
        foreach (var evictee in evictees)
        {
            _logger.LogDebug($"Evicting: {evictee}...");
            _cache.Remove(evictee, out _);
        }

        var earliest = _cache.Values
            .Where(entry => !entry.Updating)
            .OrderBy(entry => entry.Expiration)
            .FirstOrDefault();

        if (earliest == null || earliest.Expiration > DateTime.Now)
        {
            var timeout = (Int32) (earliest?.Expiration.Subtract(DateTime.Now).TotalMilliseconds ?? -1);
            _logger.LogDebug($"Waiting {timeout}ms for next expiry...");
            _governor.WaitOne(timeout);
            return;
        }

        try
        {
            _logger.LogDebug($"Updating cache for: {earliest.Key}...");
            earliest.Updating = true;
            earliest.Value = earliest.ValueFactory(dataContext);
            earliest.Expiration = earliest.ExpirySeconds > 0
                ? DateTime.Now.AddSeconds(earliest.ExpirySeconds)
                : DateTime.MaxValue;
            _governor.Set();
        }
        finally
        {
            earliest.Updating = false;
        }
    }

    public T GetOrCreate<T>(String key, Func<IDataContext, T> factory, Int32 expirySeconds = -1) where T : class 
    {
        var success = _cache.TryGetValue(key, out var entry);

        if (success && entry.Value != null)
        {
            entry.LastAccessed = DateTime.Now;
            return (T) entry.Value;
        }

        if (entry == null)
        {
            _logger.LogDebug($"Adding new entry to the cache: {key}...");
            entry = new CacheEntry
            {
                Key = key,
                Expiration = DateTime.MinValue,
                ExpirySeconds = expirySeconds,
                LastAccessed = DateTime.Now,
                ValueFactory = factory
            };

            _cache.TryAdd(key, entry);

            _governor.Set();
        }

        while (entry.Value == null)
        {
            _logger.LogDebug($"Waiting for 1st time cache update: {entry.Key}...");
            _governor.WaitOne();
        }

        return (T)entry.Value;
    }
}

3. The DataContext class is then created like so. Using Dapper for example to retrieve the data from the database:

public class DataContext : DbContext, IDataContext
{
    private readonly IOptions<Settings> _settings;
    private String _databaseServer;

    public DataContext(IOptions<Settings> settings)
    {
        _settings = settings;
    }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        base.OnConfiguring(optionsBuilder);

        optionsBuilder.UseSqlServer(_settings.Value.ConnectionString);
    }

    public IEnumerable<OrderInfo> GetOrders(DateTime date)
    {
        return Database.GetDbConnection().Query<OrderInfo>(
            $"SchemaName.usp_GetOrders",
            new {Date = date},
            commandType: CommandType.StoredProcedure);
    }
}

4. In the controllers the ICache is injected and used as follows:

    [HttpGet]
    [Route("Orders/{date}")]
    public IActionResult GetOrders(DateTime date)
    {
        var result = _cache.GetOrCreate(
            $"GetOrders_{date:yyyyMMdd}", 
            context => context.GetOrders(date),
            date.Date < DateTime.Today ? Cache.CacheIndefinitely : 20);

        return Ok(result);
    }

5. Finally register the classes as a Singleton in the DI setup

        services.AddOptions();
        services.Configure<Settings>(Configuration);
        services.AddLogging();
        services.AddDbContext<DataContext>();
        services.AddSingleton<ICache, Cache>();
        services.AddSingleton<IHostedService, CacheUpdateService>();

Answer 2

I created a similar IHostedService based project https://github.com/dpatekar/CacheAnt Take a look, it is very simple to use. It is available as a NuGet package also.