C# HttpWebRequest - send huge stream to server

Question

UPD: It turns out (from the code here https://source.dot.net/#System.Net.Requests/System/Net/HttpWebRequest.cs,1135 ) that HttpWebRequest does not support streaming, it collects data in memory and sends only on GetResponse.

So, will look into other options.

Original question

I have really huge stream, of unknown length (like "data from sensors"), that needs to be sent to server over HTTPS.

I am trying to use HttpWebRequest and send data using stream it provides.

Consdering the code below ("simulation" of large data to be sent), I expected HttpWebRequest to start sending data long before r.GetResponse().

And either stream the data, or throw an Exception if the https://some.url/endpointis unavailable, or has wrong certificate, or otherwise data cannot be sent.

However, what I get is exception that "Stream was too long". Server never gets called.

What I am dowing wrong? Should I somehow explicitly tell to HttpWebRequest to start communication?

The code:

static void Main(string[] args)
        {
            byte[] buff = new byte[64 * 1024];
            try
            {
                HttpWebRequest r = WebRequest.CreateHttp("https://some.url/endpoint");
                r.Method = "POST";
                r.AllowWriteStreamBuffering = false; // please do not do bufferring
                r.ContentType = "application/octet-stream";
                r.SendChunked = true;
                using Stream stream = r.GetRequestStream();
                for (int i = 0; i < 1000000000; i++) // really, really large stream
                {
                    stream.Write(buff, 0, buff.Length);
                }
                var resp = r.GetResponse();
                // ...
            } catch (Exception e)
            {
                Console.WriteLine($"Exception: {e.Message}"); // Stream was too long.
            }

        }

As requested in comments by @Jimi, full code snippet, .Net Core 3.1:

using System;
using System.IO;
using System.Net;

namespace HttpWebRequestConsoleTest
{
    class ConsoleTest
    {

        static void Main(string[] args)
        {
            byte[] buff = new byte[64 * 1024];
            try
            {
                HttpWebRequest r = WebRequest.CreateHttp("https://some.invalid.url/endpoint");
                r.Method = "POST";
                r.AllowWriteStreamBuffering = false; // please do not bufferring
                r.ContentType = "application/octet-stream";
                r.SendChunked = true;
                using Stream stream = r.GetRequestStream();
                for (int i = 0; i < 1000000000; i++) // really, really large stream
                {
                    stream.Write(buff, 0, buff.Length);
                    //Thread.Sleep(10);
                }
                var resp = r.GetResponse();
                // ...
            } catch (Exception e)
            {
                Console.WriteLine($"Exception: {e.Message}"); // Stream was too long.
            }

        }
    }
}

Answer 1

Here is an example of how I've done it in the past.

When you want to send data where length is unknown, you have to send it chunked. There is no other way.

To test this, I set up a new ASP.NET Core 3.1 project and added this controller:

using Microsoft.AspNetCore.Mvc;
using System.Threading.Tasks;
using WebApplication1.Utilities;

namespace WebApplication1.Controllers
{
    [ApiController, Produces("application/json"), Route("[controller]")]
    public sealed class StreamingController : ControllerBase
    {
        [HttpPost, Route("consumeStream"), DisableRequestSizeLimit]
        public async Task<IActionResult> ConsumeStreamAsync()
        {
            using (var ns = new NullStream())
            {
                await Request.Body.CopyToAsync(ns);
                return Ok(new { ns.Length });
            }
        }
    }
}

In the above example, NullStream is an object that inherits from System.IO.Stream and simply tosses out data that is written to it, but keeps track of the length.

Here is the client-side code:

using System;
using System.IO;
using System.Net;
using System.Threading.Tasks;

namespace ConsoleApp1
{
    class Program
    {
        static async Task Main()
        {
            var req = (HttpWebRequest)WebRequest.Create("https://localhost:44366/Streaming/consumeStream");
            req.Method = "POST";
            req.SendChunked = true;
            req.AllowWriteStreamBuffering = false;
            req.AllowReadStreamBuffering = false;
            req.ContentType = "application/octet-stream";

            var buf = new byte[64 * 1024];
            using(var s = await req.GetRequestStreamAsync())
            {
                for(long i = 0; i < 100_000_000; ++i)
                {
                    await s.WriteAsync(buf, 0, buf.Length);

                    if ((i & 10000) == 0)
                    {
                        Console.WriteLine($"Wrote {i * buf.Length:n0} bytes...");
                    }
                }
            }

            using (var resp = await req.GetResponseAsync())
            using (var s = resp.GetResponseStream())
            using (var sr = new StreamReader(s))
            {
                Console.WriteLine(await sr.ReadToEndAsync());
            }

            Console.WriteLine("Finished");
            Console.ReadKey(true);
        }
    }
}

For completeness' sake, here is NullStream:

using System;
using System.IO;

namespace WebApplication1.Utilities
{
    public sealed class NullStream : Stream
    {
        private long _length;
        public override bool CanRead => false;
        public override bool CanSeek => false;
        public override bool CanWrite => true;
        public override long Length => _length;
        public override long Position
        { 
            get => throw new NotImplementedException();
            set => throw new NotImplementedException();
        }
        public override void Flush() { }
        public override int Read(byte[] buffer, int offset, int count) =>
            throw new NotImplementedException();
        public override long Seek(long offset, SeekOrigin origin) =>
            throw new NotImplementedException();
        public override void SetLength(long value) =>
            _length = value;
        public override void Write(byte[] buffer, int offset, int count)
        {
            _length += count;
            System.Diagnostics.Debug.WriteLine($"Write: {count:n0}; Length: {_length:n0}");
        }
    }
}

Example of the output:

...
Write: 65,536; Length: 4,514,222,022
Write: 57,344; Length: 4,514,279,366
Write: 65,536; Length: 4,514,344,902
Write: 65,536; Length: 4,514,410,438
Write: 65,536; Length: 4,514,475,974
Write: 57,344; Length: 4,514,533,318
Write: 61,440; Length: 4,514,594,758
Write: 65,536; Length: 4,514,660,294
Write: 65,536; Length: 4,514,725,830
Write: 65,536; Length: 4,514,791,366
Write: 65,536; Length: 4,514,856,902
...