Asp.Net Core 5 API Web upload large file in streamed mode like WCF

Question

I need to upload a large file (7GB) in streaming mode (one piece at a time) on my web server made in asp .net core 5.

Configuration Server side:

public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>().UseKestrel(options =>
                    {
                        options.Limits.MaxRequestBodySize = long.MaxValue;
                       
                    });
                });
    }

Controller in Server Side:

[HttpPost]
[RequestFormLimits(MultipartBodyLengthLimit = Int32.MaxValue)]
public async Task PostStream(IFormFile formFiles)
{
    //rest of code
}

Client Side ( Desktop app )

using (var httpClient = new HttpClient())
                {
                    using (var content = new MultipartFormDataContent())
                    {
                        string fileName = @"D:\largeFile.zip";

                        FileStream fileStream = File.OpenRead(fileName);
                        HttpContent fileStreamContent = new StreamContent(fileStream);

                        fileStreamContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") { Name = "formFiles", FileName = fileName };
                        fileStreamContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");

                        content.Add(fileStreamContent);

                       
                        var response = httpClient.PostAsync("http://localhost:5000/weatherforecast", content).Result;
                        response.EnsureSuccessStatusCode();
                }

The system works up to 1 GB, with large files (7GB) the server cancels the request, I noticed that the files that are sent to the server are not streaming but upload them completely, I was expecting a progressive upload on the server as it happens on WCF. How can i send a file to the server progressively?

Answer 1

In your code [RequestFormLimits(MultipartBodyLengthLimit = Int32.MaxValue)] limits the upload to 4Gb

Change this to [RequestFormLimits(MultipartBodyLengthLimit = Int64.MaxValue)]

Also have a look at the approach given here https://code-maze.com/aspnetcore-upload-large-files/ this is the approach to use, where is save is small packets and not loaded in the server memory all at once

The critical part for me was you must disable all asp.net core model binders, after doing this Request.Body became usable.

Answer 2

I solved it but the stream must stand alone I cannot accompany the stream with additional data present in a class or variable that travels together with the stream

Client Side:

            using (var client = new HttpClient())
            {
                client.Timeout = TimeSpan.FromMinutes(16);
                string fileName = @"D:\BIGFILE7GB.zip";
                var fileStreamContent = new StreamContent(File.OpenRead(fileName));

                var response = client.PostAsync("http://localhost:5000/weatherforecast/upload", fileStreamContent).Result;
                response.EnsureSuccessStatusCode();
            }

Server Side

public async Task<IActionResult> Upload()
        {
           //stream are here!!!!! 
           //Request.Body.Read() <------ magic

           // routine to save file
           while ((len = await Request.Body.ReadAsync(buffer, 0, buffer.Length)) > 0)
            {
                file.Write(buffer, 0, len);
            }
            

            await Task.Delay(1);
            return Ok();
        }

The solution is to choose whether to send only the streaming (so that the control clicks immediately and not only after the file has been completely loaded) or only the form data. The question is is there a way to send the stream and also form data together? Always respecting the constraint of immediately starting the action of the server-side controller by starting the streaming stream and not when the whole file has been uploaded on the client side?

a solution like this:

public async Task<IActionResult> Upload(MyClass class)
{

    Read variable from class form and read buffer in straming togheter
}

Answer 3

I managed to overcome the 7G limit simply by adding in the controller head this:

[HttpPost]
 [RequestFormLimits(ValueLengthLimit = int.MaxValue, MultipartBodyLengthLimit = Int64.MaxValue)]
public async Task PostStream(IFormFile formFiles)
{
    //rest of code
}

the following problem still remains:

I also do not understand why the upload does not start immediately but only after everything has been uploaded, I expect a progressive upload so the server-side controller clicks immediately and not only after the whole file has been uploaded

Answer 4

Because of your server-side code

[HttpPost]
[RequestFormLimits(MultipartBodyLengthLimit = Int32.MaxValue)]
public async Task PostStream(IFormFile formFiles)
{
    //rest of code
}

I assume that you haven't followed this tutorial yet. And there's too much code in the tutorial, so I will summarize what I did and give you an example below.

First, you will need a Filter to disable model binding.

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
    public void OnResourceExecuting(ResourceExecutingContext context)
    {
        if (!RequestHelper.IsMultipartContentType(context.HttpContext.Request.ContentType))
        {
            context.Result = new BadRequestObjectResult("Request is not a 'multipart' request");
        }

        var factories = context.ValueProviderFactories;
        factories.RemoveType<FormValueProviderFactory>();
        factories.RemoveType<FormFileValueProviderFactory>();
        factories.RemoveType<JQueryFormValueProviderFactory>();
    }

    public void OnResourceExecuted(ResourceExecutedContext context)
    {
    }
}

Then, the service code which handles upload request. Make sure you change it a little bit to save the file into disk instead of memory.

public class MultipartRequestService : IMultipartRequestService
{
    public async Task<(Dictionary<string, StringValues>, byte[])> GetDataFromMultiPart(MultipartReader reader, CancellationToken cancellationToken)
    {
        var formAccumulator = new KeyValueAccumulator();
        var file = new byte[0];

        MultipartSection section;
        while ((section = await reader.ReadNextSectionAsync(cancellationToken)) != null)
        {
            if (!ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var contentDisposition))
            {
                continue;
            }

            if (contentDisposition.IsFormDisposition())
            {
                formAccumulator = await AccumulateForm(formAccumulator, section, contentDisposition);
            }

            else if (contentDisposition.IsFileDisposition())
            {
                // you will want to replace all of this because it copies the file into a memory stream. We don't want that.
                await using var memoryStream = new MemoryStream();
                await section.Body.CopyToAsync(memoryStream, cancellationToken);

                file = memoryStream.ToArray();
            }
        }

        return (formAccumulator.GetResults(), file);
    }

    private Encoding GetEncoding(MultipartSection section)
    {
        var hasMediaTypeHeader = MediaTypeHeaderValue.TryParse(section.ContentType, out var mediaType);

        // UTF-7 is insecure and shouldn't be honored. UTF-8 succeeds in 
        // most cases.
        if (!hasMediaTypeHeader || Encoding.UTF7.Equals(mediaType.Encoding))
        {
            return Encoding.UTF8;
        }

        return mediaType.Encoding;
    }

    private async Task<KeyValueAccumulator> AccumulateForm(KeyValueAccumulator formAccumulator, MultipartSection section, ContentDispositionHeaderValue contentDisposition)
    {
        var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name).Value;
        using var streamReader = new StreamReader(section.Body, GetEncoding(section), true, 1024, true);
        {
            var value = await streamReader.ReadToEndAsync();
            if (string.Equals(value, "undefined", StringComparison.OrdinalIgnoreCase))
            {
                value = string.Empty;
            }
            formAccumulator.Append(key, value);

            if (formAccumulator.ValueCount > FormReader.DefaultValueCountLimit)
            {
                throw new InvalidDataException($"Form key count limit {FormReader.DefaultValueCountLimit} exceeded.");
            }
        }

        return formAccumulator;
    }
}

Finally, the controller code

[HttpPost()]
[DisableFormValueModelBinding]
public async Task<IActionResult> Upload(
    [FromServices] IMultipartRequestService multipartRequestService,
    CancellationToken cancellationToken)
{
    var reader = new MultipartReader(HttpContext.Request.GetMultipartBoundary(), HttpContext.Request.Body)
    {
        BodyLengthLimit = uploadLimit.Value.ChunkSizeLimitInBytes
    };

    _logger.LogInformation("Read data and file from uploaded request");
    var (forms, file) = await multipartRequestService.GetDataFromMultiPart(reader, cancellationToken);

    // do anything you want

    return Ok();
}