CODEWITHSUNDEEP
c#.net-coreentity-framework-corehealth-check
Michael Pedersen
Michael Pedersen

Reputation: 91

Health Check checking data from EF DB causes threading issue

I am implementing a set of health check to a .net core 3.1 application with the AspNetCore.HealthCheck nuget package. Some of the health check have to reach a EFcore database to check if some data updated by other systems are present to validate other processes have run properly.

When implementing one health check for this everything works great, but as soon as I implement the second health check which does more or less the same, with a few variants, I get a threading issue as the first call to the EF core has not completed before the next arrives.

The EF core code from the repository

public async Task<IEnumerable<EstateModel>> ListEstates(string customerId)
    {
        try
        {
            var estates = _productDbContext.Estates.AsNoTracking().Where(p => p.CustomerId == customerId)
                .Include(e => e.Meters)
                .ThenInclude(m => m.Counters)
                .Include(e => e.Installations);
            var entities = await estates.ToListAsync().ConfigureAwait(false);

            return _mapper.Map<List<EstateModel>>(entities);
        }
        catch (Exception ex)
        {
            Log.Error($"Error listing estate by customer: {customerId}", ex);
        }

        return null;
    }

An example of the health check

public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = new CancellationToken())
    {
        var configs = new List<ConsumptionHealthCheckConfig>();
        _configuration.GetSection("HealthCheckSettings:GetConsumptionGas").Bind(configs);

        foreach (var config in configs)
        {
            try
            {
                return await _healthService.CheckConsumptionHealth(config, false, false);
            }
            catch(Exception ex)
            {
                return new HealthCheckResult(HealthStatus.Unhealthy, $"An error occurred while getting consumption for {config.Detailed.InstallationNumber} {ex}", ex);
            }
        }

        return new HealthCheckResult(HealthStatus.Healthy);
    }

The healthservice method

public async Task<HealthCheckResult> CheckConsumptionHealth(ConsumptionHealthCheckConfig config, bool isWater, bool isHeating)
    {
        if ((config.Detailed?.InstallationNumber ?? 0) != 0 && (config.Detailed?.MeterNumber ?? 0) != 0)
        {
            var estates = await _estateService.GetEstates(config.Detailed.CustomerNo);
Rest is omitted...

The AddHealthChecks in Configure services

internal static void Configure(IConfiguration configuration, IServiceCollection services)
    {
        services.AddHealthChecks()
            //Consumption
            .AddCheck<GetConsumptionElectricityHealthCheck>("Consumption Electricity", failureStatus: HealthStatus.Unhealthy, tags: new[] {"Consumption"})
            .AddCheck<GetConsumptionWaterHealthCheck>("Consumption Water", failureStatus: HealthStatus.Unhealthy, tags: new[] {"Consumption"})

The exception that I'm getting is

A second operation started on this context before a previous operation completed. This is usually caused by different threads using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.

and when looking at the link provided, it states that I should always await any calls the database immediately, which we clearly do.

I have tried moving the GetEstates part to the health check itself instead of my service, but then I get an issue where trying to reach the database while it is being configured.

So my problem arrises when these consumption health checks all reach the EF core at the same time, but I cannot see how to circumvent that from happening as there are no apparent options to tell the health checks to run in sequence or if I implement a butt-ugly Thread.Sleep and as far as I know, it shouldn't be necessary to implement thread locking on top of EF Core or am I incorrect?

Any help will be greatly appreciated!

Upvotes: 0

Views: 1472

Answers (1)

Jeremy Lakeman
Jeremy Lakeman

Reputation: 11163

As discussed in this issue, all health checks use the same service scope and run in parallel. I'd recommend that you create a new service scope inside any health check that accesses your DbContext.

public virtual async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default(CancellationToken))
{
    using var scope = serviceProvider.CreateScope();
    var healthService = scope.ServiceProvider.GetRequiredService<...>();
    ...
}

Upvotes: 1

Related Questions