Implement Downloading Large files > 2gb in ASP.NET Core

Question

This is my code:

var net = new System.Net.WebClient();
var data = net.DownloadData(zipPath);
var content = new MemoryStream(data);
var contentType = "APPLICATION/octet-stream";
var fileName = zipPath.Split('\\')[zipPath.Split('\\').Length -1];
Response.Cookies.Append("download", "finished");
return File(content, contentType, fileName);

However, DownloadData(zipPath) gives error as WebException: "The message length limit was exceeded" it seems like it can't read more than 2GB of size, I searched and it needs to edit some property to -1 of an object which i didn't use in my code.. https://social.msdn.microsoft.com/Forums/en-US/88d0c0bb-ec86-435d-9d2a-c5ec821e9a79/httpwebresponse-maximum-response-header-size-response-over-64k-problem?forum=winappswithcsharp

and I have tried this aswell:

HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
var stream = new FileStream(zipPath, FileMode.Open, FileAccess.Read);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream");

Gave this exception: System.IO.IOException: 'The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.'

I don't know what to do anymore.. any advice? In the end i'm trying to return File(byte[]) so I could download that file from the server.

Answer 1

Thanks for everyone who tried helping me, I have found the solution myself after digging hard in the documentation and re checking answers, combining ideas, etc.. You got two solutions:

1- Make a class that implements Stream as a HugeMemoryStream, Which of course you can find here:

class HugeMemoryStream : System.IO.Stream
{
    #region Fields

    private const int PAGE_SIZE = 1024000000;
    private const int ALLOC_STEP = 1024;

    private byte[][] _streamBuffers;

    private int _pageCount = 0;
    private long _allocatedBytes = 0;

    private long _position = 0;
    private long _length = 0;

    #endregion Fields

    #region Internals

    private int GetPageCount(long length)
    {
        int pageCount = (int)(length / PAGE_SIZE) + 1;

        if ((length % PAGE_SIZE) == 0)
            pageCount--;

        return pageCount;
    }

    private void ExtendPages()
    {
        if (_streamBuffers == null)
        {
            _streamBuffers = new byte[ALLOC_STEP][];
        }
        else
        {
            byte[][] streamBuffers = new byte[_streamBuffers.Length + ALLOC_STEP][];

            Array.Copy(_streamBuffers, streamBuffers, _streamBuffers.Length);

            _streamBuffers = streamBuffers;
        }

        _pageCount = _streamBuffers.Length;
    }

    private void AllocSpaceIfNeeded(long value)
    {
        if (value < 0)
            throw new InvalidOperationException("AllocSpaceIfNeeded < 0");

        if (value == 0)
            return;

        int currentPageCount = GetPageCount(_allocatedBytes);
        int neededPageCount = GetPageCount(value);

        while (currentPageCount < neededPageCount)
        {
            if (currentPageCount == _pageCount)
                ExtendPages();

            _streamBuffers[currentPageCount++] = new byte[PAGE_SIZE];
        }

        _allocatedBytes = (long)currentPageCount * PAGE_SIZE;

        value = Math.Max(value, _length);

        if (_position > (_length = value))
            _position = _length;
    }

    #endregion Internals

    #region Stream

    public override bool CanRead => true;

    public override bool CanSeek => true;

    public override bool CanWrite => true;

    public override long Length => _length;

    public override long Position
    {
        get { return _position; }
        set
        {
            if (value > _length)
                throw new InvalidOperationException("Position > Length");
            else if (value < 0)
                throw new InvalidOperationException("Position < 0");
            else
                _position = value;
        }
    }

    public override void Flush() { }

    public override int Read(byte[] buffer, int offset, int count)
    {
        int currentPage = (int)(_position / PAGE_SIZE);
        int currentOffset = (int)(_position % PAGE_SIZE);
        int currentLength = PAGE_SIZE - currentOffset;

        long startPosition = _position;

        if (startPosition + count > _length)
            count = (int)(_length - startPosition);

        while (count != 0 && _position < _length)
        {
            if (currentLength > count)
                currentLength = count;

            Array.Copy(_streamBuffers[currentPage++], currentOffset, buffer, offset, currentLength);

            offset += currentLength;
            _position += currentLength;
            count -= currentLength;

            currentOffset = 0;
            currentLength = PAGE_SIZE;
        }

        return (int)(_position - startPosition);
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        switch (origin)
        {
            case SeekOrigin.Begin:
                break;

            case SeekOrigin.Current:
                offset += _position;
                break;

            case SeekOrigin.End:
                offset = _length - offset;
                break;

            default:
                throw new ArgumentOutOfRangeException("origin");
        }

        return Position = offset;
    }

    public override void SetLength(long value)
    {
        if (value < 0)
            throw new InvalidOperationException("SetLength < 0");

        if (value == 0)
        {
            _streamBuffers = null;
            _allocatedBytes = _position = _length = 0;
            _pageCount = 0;
            return;
        }

        int currentPageCount = GetPageCount(_allocatedBytes);
        int neededPageCount = GetPageCount(value);

        // Removes unused buffers if decreasing stream length
        while (currentPageCount > neededPageCount)
            _streamBuffers[--currentPageCount] = null;

        AllocSpaceIfNeeded(value);

        if (_position > (_length = value))
            _position = _length;
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        int currentPage = (int)(_position / PAGE_SIZE);
        int currentOffset = (int)(_position % PAGE_SIZE);
        int currentLength = PAGE_SIZE - currentOffset;

        long startPosition = _position;

        AllocSpaceIfNeeded(_position + count);

        while (count != 0)
        {
            if (currentLength > count)
                currentLength = count;

            Array.Copy(buffer, offset, _streamBuffers[currentPage++], currentOffset, currentLength);

            offset += currentLength;
            _position += currentLength;
            count -= currentLength;

            currentOffset = 0;
            currentLength = PAGE_SIZE;
        }
    }

    #endregion Stream
}

2- or just push the file to the clients by simply returning the file from HDD (this might be slow if you want faster transfer use better storage units or move them to the blazing fast RAM..) through the controller and to the view. using this specific return type:

return new PhysicalFileResult("Directory Containing File", 
"application/octet-stream") 
{ FileDownloadName = "Your file name + extension, for example: test.txt or test.zip etc.." };

It was pain in the ass but worth it since no one is really answering this question online :)

Answer 2

Trying to return items > 2GB from the same I/O call is never gonna end well. At no point ever do you want a single byte[] with then entire contents of the item. Instead, when working with such large items you want to get a stream, where you can read and buffer only small segments at a time. You can use WebClient.GetWebRequest() and WebClient.GetWebResponse() to accomplish this.