CODEWITHSUNDEEP
asp.net-core.net-core
bubbleking
bubbleking

Reputation: 3601

Design Minimal API and use HttpClient to post a file to it

I have a legacy system interfacing issue that my team has elected to solve by standing up a .NET 7 Minimal API which needs to accept a file upload. It should work for small and large files (let's say at least 500 MiB). The API will be called from a legacy system using HttpClient in a .NET Framework 4.7.1 app.

I can't quite seem to figure out how to design the signature of the Minimal API and how to call it with HttpClient in a way that totally works. It's something I've been hacking at on and off for several days, and haven't documented all of my approaches, but suffice it to say there have been varying results involving, among other things:

On the Minimal API side, I've tried all sorts of things in the signature (IFormFile, Stream, PipeReader, HttpRequest). On the calling side, I've tried several approaches (messing with headers, using the Flurl library, various content encodings and MIME types, multipart, etc).

This seems like it should be dead simple, so I'm trying to wipe the slate clean here, start with an example of something that partially works, and hope someone might be able to illuminate the path forward for me.

Example of Minimal API:

// IDocumentStorageManager is an injected dependency that takes an int and a Stream and returns a string of the newly uploaded file's URI

app.MapPost(
    "DocumentStorage/CreateDocument2/{documentId:int}",
    async (PipeReader pipeReader, int documentId, IDocumentStorageManager documentStorageManager) =>
    {
        using var ms = new MemoryStream();
        await pipeReader.CopyToAsync(ms);
        ms.Position = 0;
        return await documentStorageManager.CreateDocument(documentId, ms);
    });

Call the Minimal API using HttpClient:

    // filePath is the path on local disk, uri is the Minimal API's URI

    private static async Task<string> UploadWithHttpClient2(string filePath, string uri)
    {
        var fileStream = File.Open(filePath, FileMode.Open);
        var content = new StreamContent(fileStream);
        var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri);
        var httpClient = new HttpClient();
        
        httpRequestMessage.Content = content;
        httpClient.Timeout = TimeSpan.FromMinutes(5);

        var result = await httpClient.SendAsync(httpRequestMessage);

        return await result.Content.ReadAsStringAsync();
    }

In the particular example above, a small (6 bytes) .txt file is uploaded without issue. However, a large (619 MiB) .tif file runs into problems on the call to httpClient.SendAsync which results in the following set of nested Exceptions:

System.Net.Http.HttpRequestException - "Error while copying content to a stream."
  System.IO.IOException - "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.."
    System.Net.Sockets.SocketException - "An existing connection was forcibly closed by the remote host."

What's a decent way of writing a Minimal API and calling it with HttpClient that will work for small and large files?

Upvotes: 1

Views: 1729

Answers (2)

davidfowl
davidfowl

Reputation: 38764

This answer is good but the RequestSizeLimit filter doesn't work for minimal APIs, it's an MVC filter. You can use the IHttpMaxRequestBodySizeFeature to limit the size (assuming you're not running on IIS). Also, I made a change to accept the body as a Stream. This avoids the memory stream copy before calling the CreateDocument API:

app.MapPost(
    "DocumentStorage/CreateDocument2/{documentId:int}",
    async (Stream stream, int documentId, IDocumentStorageManager documentStorageManager) =>
    {
        return await documentStorageManager.CreateDocument(documentId, stream);
    })
    .AddEndpointFilter((context, next) =>
    {
        const int MaxBytes = 1024 * 1024 * 1024;

        var maxRequestBodySizeFeature = context.HttpContext.Features.Get<IHttpMaxRequestBodySizeFeature>();

        if (maxRequestBodySizeFeature is not null and { IsReadOnly: true })
        {
            maxRequestBodySizeFeature.MaxRequestBodySize = MaxBytes;
        }

        return next(context);
    });

If you're running on IIS then https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/requestlimits/#configuration

Upvotes: 0

Wolfspirit
Wolfspirit

Reputation: 778

Kestrel allows uploading 30MB per default.
To upload larger files via kestrel you might need to increase the max size limit. This can be done by adding the "RequestSizeLimit" attribute. So for example for 1GB:

app.MapPost(
    "DocumentStorage/CreateDocument2/{documentId:int}",
    [RequestSizeLimit(1_000_000_000)] async (PipeReader pipeReader, int documentId) =>
    {
        using var ms = new MemoryStream();
        await pipeReader.CopyToAsync(ms);
        ms.Position = 0;
        return "";
    });

You can also remove the size limit globally by setting

builder.WebHost.UseKestrel(o => o.Limits.MaxRequestBodySize = null);

Upvotes: 4

Related Questions