DbUpdateConcurrencyException in Outbox MassTransit

Question

We are currently using MassTransit and MassTransit embedded Outbox with generally successful results in our development and testing environments. However, when we transition to the live environment, we encounter some unusual behavior. Specifically, as the volume of messages increases, we start receiving a significant number of Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException errors in the live environment.

MassTransit.EntityFrameworkCore: 8.0.16

Microsoft.EntityFrameworkCore.SqlServer : 7.0.8,

net7.0

MT-Fault-Message:   The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
MT-Fault-StackTrace:    at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyExceptionAsync(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetAsync(Int32 startCommandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at MassTransit.EntityFrameworkCoreIntegration.EntityFrameworkOutboxContextFactory`1.<>c__DisplayClass6_0`1.<<Send>g__Execute|0>d.MoveNext() in /_/src/Persistence/MassTransit.EntityFrameworkCoreIntegration/EntityFrameworkCoreIntegration/EntityFrameworkOutboxContextFactory.cs:line 76
--- End of stack trace from previous location ---
at MassTransit.EntityFrameworkCoreIntegration.EntityFrameworkOutboxContextFactory`1.<>c__DisplayClass6_0`1.<<Send>g__Execute|0>d.MoveNext() in /_/src/Persistence/MassTransit.EntityFrameworkCoreIntegration/EntityFrameworkCoreIntegration/EntityFrameworkOutboxContextFactory.cs:line 104
--- End of stack trace from previous location ---
at MassTransit.EntityFrameworkCoreIntegration.EntityFrameworkOutboxContextFactory`1.<>c__DisplayClass6_0`1.<<Send>g__Execute|0>d.MoveNext() in /_/src/Persistence/MassTransit.EntityFrameworkCoreIntegration/EntityFrameworkCoreIntegration/EntityFrameworkOutboxContextFactory.cs:line 104
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass30_0`2.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at MassTransit.EntityFrameworkCoreIntegration.EntityFrameworkOutboxContextFactory`1.Send[T](ConsumeContext`1 context, OutboxConsumeOptions options, IPipe`1 next) in /_/src/Persistence/MassTransit.EntityFrameworkCoreIntegration/EntityFrameworkCoreIntegration/EntityFrameworkOutboxContextFactory.cs:line 113
at MassTransit.Middleware.OutboxConsumeFilter`2.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/OutboxConsumeFilter.cs:line 41
at MassTransit.Middleware.OutboxConsumeFilter`2.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/OutboxConsumeFilter.cs:line 41
at MassTransit.Middleware.RetryFilter`1.Attempt(TContext context, RetryContext`1 retryContext, IPipe`1 next) in /_/src/MassTransit/Middleware/RetryFilter.cs:line 158
at MassTransit.Middleware.RetryFilter`1.Attempt(TContext context, RetryContext`1 retryContext, IPipe`1 next) in /_/src/MassTransit/Middleware/RetryFilter.cs:line 238
at MassTransit.Middleware.RetryFilter`1.MassTransit.IFilter<TContext>.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/RetryFilter.cs:line 128
at MassTransit.Middleware.RetryFilter`1.MassTransit.IFilter<TContext>.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/RetryFilter.cs:line 47
at MassTransit.Middleware.RetryFilter`1.MassTransit.IFilter<TContext>.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/RetryFilter.cs:line 118
at MassTransit.Middleware.RetryFilter`1.MassTransit.IFilter<TContext>.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/RetryFilter.cs:line 47
at MassTransit.Middleware.RetryFilter`1.MassTransit.IFilter<TContext>.Send(TContext context, IPipe`1 next) in /_/src/MassTransit/Middleware/RetryFilter.cs:line 118

To address this issue, I attempted to handle the exception by configuring our message retry mechanism as follows:

endpointConfigurator.UseMessageRetry(r =>
{
    r.Handle<DbUpdateConcurrencyException>();
    r.None();
});

endpointConfigurator.UseMessageRetry(r =>
{
    r.Ignore<DbUpdateConcurrencyException>();
    r.Intervals(
        TimeSpan.FromMilliseconds(10),
        TimeSpan.FromMilliseconds(100),
        TimeSpan.FromSeconds(1)
    );
});

Unfortunately, even with this setup, there are instances where DbUpdateConcurrencyException errors still end up in the error queue. What's particularly puzzling is that when these exceptions occur in other queues that are not explicitly configured to ignore them, they also affect the queue that is supposed to ignore them.

Additionally, I attempted to resolve the issue by setting the service lifetime of the DbContext to Transient, but this change did not produce the desired results.

We are currently seeking assistance and advice on how to address this issue and ensure the reliable operation of our application in the live environment.

Answer 1

You should always retry DbUpdateConcurrencyException, and yet, you are going to great lengths to ensure you do not retry them. Your retry configuration is wrong.

endpointConfigurator.UseMessageRetry(r =>
{
    r.Handle<DbUpdateConcurrencyException>();
    r.Intervals(
        TimeSpan.FromMilliseconds(10),
        TimeSpan.FromMilliseconds(100),
        TimeSpan.FromSeconds(1)
    );
});