How to resolve "Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly"?

Question

While doing API Performance Testing [e.g. No of Threads/Users : 50 with Loop Count : 10], 5 to 8 % samples with POST methods are failing due to below exception :

**Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.**
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.PumpAsync()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)

Could you please suggest how i can fix this ?

NOTE :I correctly implemented async/await in POST method.This is happening to POST method only and out of 500 samples only 20 to 40 samples are failing per POST method. Providing same inputs while testing.

Answer 1

Requests should be resolved fast, and this is a clear warning, that they are not. Exception is, that we are debugging/single-stepping the service, in which case I would prefer more tolerant request limits. In live environments, DDoS and even heavy load can render a system or service unstable very quickly, if limits are too high, because a growing request stack can overwhelm the system.

I made a cache service built on Kestrel and I found two scenarios which could cause this error:

1. The service received a massive amounts of items for a write operation on a quite busy DB table. This happened due to my own poor design. To resolve: either to optimize the update flow or create a per-task based update handler.

2. Service requests have stalled due to an instability, suspension or shut-down. (f.ex. TCP connections may not always shut down immediately using a static HttpClient, it sometimes fails to exit gracefully on shut-down)

Code below is from a NET 7.0 based kestrel service

// Limits can also be applied within appsettings.json, 
appBuilder.WebHost.ConfigureKestrel(serverOptions =>
{
    serverOptions.Limits.MaxConcurrentConnections = 100;
    serverOptions.Limits.MaxConcurrentUpgradedConnections = 100;

#if !DEBUG
    // Live: Demand higher request data-rate limits and low grace period, services can be unreliable.
    // May cause requests to stall resulting in BadHttpRequestException.
    // Catch them in your error handler
    serverOptions.Limits.MinRequestBodyDataRate = new MinDataRate(
        bytesPerSecond: 100,
        gracePeriod: TimeSpan.FromSeconds(5)
    );
#endif
});

Add error handler to app

// global error handler
app.UseMiddleware<ErrorHandlerMiddleware>();

Error handler:

public class ErrorHandlerMiddleware
{
    private readonly RequestDelegate _next;

    public ErrorHandlerMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext context)
    {
        try
        {
            await _next(context);
        }
#if !DEBUG
        catch (Microsoft.AspNetCore.Connections.ConnectionResetException)
        {
            // Not of interest - client has ended the connection prematurely. Happens all the time.
            return;
        }
#endif
        catch (BadHttpRequestException bad) when (bad.Message.Contains("data arriving too slowly"))
        {
#if DEBUG
            // The original exception lacks insight.
            throw new BadHttpRequestException($"Request data arriving too slowly: {context.Request.Path}");
#endif
        }
    }
}

Answer 2

You could setKestrelServerLimits.MinRequestBodyDataRate property to null in your Program.cs as below:

.UseKestrel(opt =>
{
    opt.Limits.MinRequestBodyDataRate = null;
});

Reference:

KestrelServerLimits.MinRequestBodyDataRate Property