Getting DbUpdateConcurrencyException, but only in production. Cannot reproduce in dev

Question

I'm working on a Quartz.NET hosted job as part of a Blazor Server web application. This job downloads a list of products from a warehouse, and caches them in a local DbContext. In order to optimize performance due to the large number of entries, I'm inserting the new items in batches of 1000 (with DbSet.AddRange()), and after each batch I submit the changes with database.SaveChangesAsync().

The following code is a simplified version of this process:

public async Task Execute(IJobExecutionContext context)
{
    // databaseFactory is injected as a dependency (IDbContextFactory<AppDatabase>)
    await using var database = await databaseFactory.CreateDbContextAsync();

    int page = 0;
    while(true)
    {
        List<WarehouseEntry> dbItems = warehouse.QueryPage(page, 1000); // Retrieves 1000 entries
        if(dbItems.Count == 0) 
            break;

        database.ProductWarehouseCache.AddRange(dbItems.Select(p => new CachedWarehouseProduct {
            Sku = p.Name,
            Barcode = p.Barcode,
            Stock = p.Stock,
            Price = p.Price
        }));

        await database.SaveChangesAsync(); // <---- DbUpdateConcurrencyException here
        page++;
    }
}

Note that there is absolutely no concurrency on the code above. The IJob class has the [DisallowConcurrentExecution] attribute meaning that even if I accidentally trigger this procedure multiple times simultaneously, only one instance will be executing at any given time, so despite the exception message, this is not a concurrency issue. It's also important to note that nothing else is updating/querying the database while this code is running.

This works as intended on my local development machine. However when I tried to deploy the application to a production server for the first time, I've found that this specific part of the code fails with a DbUpdateConcurrencyException. Normally with an exception like this, I would look for concurrency issues, or DbContexts that are used by multiple threads at the same time, or aren't disposed properly. However, as I have explained above, this is not the case here.

The following is the full exception message:

Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: 
The database operation was expected to affect 1 row(s), but actually affected 0 row(s); 
data may have been modified or deleted since entities were loaded. 
See http://go.microsoft.com/fwlink/?LinkId=527962 for information on 
understanding and handling optimistic concurrency exceptions.

What could be causing an exception like this, when there is no concurrency whatsoever? And what could cause this to only happen on the production server, but never in the development workspace.

Additional information:

• dotnet 6
• EF Core 6.0.6
• Local/Dev Database: MySQL 8.0.31
• Local/Dev OS: Windows 11
• Remote/Prod Database+OS: MySQL 8.0.30-0ubuntu0.20.04.2

Answer 1

I have fixed the issue. I was using DataGrip's built-in import/export tools to clone my database's DDL from local dev DB to remote prod DB. Apparently these tools don't replicate the DDL exactly as they should, which leads to EF core throwing random unexpected errors such as this.

To fix it, I rewrote my deployment pipeline to use the dotnet ef migrations script --idempotent script to generate an .sql file that automatically applies any missing migrations to the production database. By using this dotnet tool, I am no longer getting the exception.