"Too many open files" and request buffering

Question

We have an ASP.NET Core 5 app running in a Docker container in AWS.

When it receives a burst of HTTP POST requests, I get this error in the logs:

System.IO.IOException: Too many open files
    at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
    at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.CreateTempFile()
    at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
    at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
    at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
    at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
    at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value, Object container)
    at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c_DisplayClass0_0.<<CreateBinderDelegate>g_Bind|0>d.MoveNext()

Going through the exception trace above we can see that:

1. This is happening in the MVC middleware (this is, before it gets to our own Controller code)
2. This is happening when the FileBufferingReadStream is trying to create a new file to disk when reading the input stream (see source code here). This gets called when you enable buffering in your solution.

What we have so far is: something is enabling buffering in the solution, which causes us to write to disk every time we get a request with a long body (happens most of the time). However, I have removed all code that calls EnableBuffering from my solution. Is there any code in a library/framework that can be enabling buffering from us? Is this controlled by some environmental variable that could be part of the environment this is running in?

Edit:

This is my Program.cs class (I have omitted some stuff for privacy):

public static async Task Main(string[] args)
{
    var webHost = CreateHostBuilder(args).Build();
    await webHost.RunAsync();
}

public static IHostBuilder CreateHostBuilder(string[] args)
{
    return Host.CreateDefaultBuilder(args)
        .ConfigureLogging(logging =>
        {
            logging.ClearProviders();
            logging.AddConsole();
            logging.AddDebug();
            logging.SetMinimumLevel(LogLevel.Information);
        })
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
            webBuilder.CaptureStartupErrors(false);
            webBuilder.UseSentry();
        })
        .ConfigureServices(services => { });
}

and this is my Startup.cs class:

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    private IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.Configure<Config>(Configuration.GetSection("AppSettings"));

        services.AddApiVersioning(version =>
        {
            version.DefaultApiVersion = new ApiVersion(1, 0);
            version.AssumeDefaultVersionWhenUnspecified = true;
            version.ReportApiVersions = true;
        });

        services.AddLogging();
        
        //... DI config here

        services.AddControllers();
        services.AddHealthChecks();

        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo {Title = "MyApp", Version = "v1"});
        });
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IMainIndexer elastic)
    {
        app.UseRouting();
        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
            endpoints.MapHealthChecks("/healthcheck", new HealthCheckOptions
            {
                Predicate = _ => true,
                ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
            });
            endpoints.MapGet("/",
                async context =>
                {
                    await context.Response.WriteAsync(
                        "<html><body>Welcome to MyApp</body></html>");
                });
        });
    }
}

We have tried some workarounds, like using a different Docker base image and using Newtonsoft instead of System.Text and we still get the same issue. Am I missing something?

Answer 1

As mentioned in the question, the issue is caused by something in the middleware enabling request buffering.

We've found out that this was caused by Sentry.

You can see here that Sentry calls EnableBuffering().

When we removed this line from Program.cs

WebBuilder.UseSentry();

and ran the same load again, the errors were gone.