Using PushStreamContent to upload from an HTTPClient

Question

I would like to upload a large amount of data to a web server from a client machine. I jumped right to PushStreamContent so I could write directly to the stream, as the results vary in size and can be rather large.

The flow is as follows:

User runs query > Reader Ready Event Fires > Begin Upload

Once the ready event is fired, the listener picks it up and iterates over the result set, uploading the data as a multipart form:

Console.WriteLine("Query ready, uploading");
        byte[] buffer = new byte[1024], form = new byte[200];
        int offset = 0, byteCount = 0;
        StringBuilder rowBuilder = new StringBuilder();
        string builderS;
        var content = new PushStreamContent(async (stream, httpContent, transportContext) =>
        //using (System.IO.Stream stream = new System.IO.FileStream("test.txt", System.IO.FileMode.OpenOrCreate))
        {
            int bytes = 0;
            string boundary = createFormBoundary();
            httpContent.Headers.Remove("Content-Type");
            httpContent.Headers.TryAddWithoutValidation("Content-Type", "multipart/form-data; boundary=" + boundary);
            await stream.WriteAsync(form, 0, form.Length);
            form = System.Text.Encoding.UTF8.GetBytes(createFormElement(boundary, "file"));
            await stream.WriteAsync(form, 0, form.Length);
            await Task.Run(async () =>
            {
                foreach (var row in rows)
                {
                    for (int i = 0; i < row.Length; i++)
                    {
                        rowBuilder.Append(row[i].Value);
                        if (i + 1 < row.Length)
                            rowBuilder.Append(',');
                        else
                        {
                            rowBuilder.Append("\r\n");
                        }
                    }
                    builderS = rowBuilder.ToString();
                    rowBuilder.Clear();
                    byteCount = System.Text.Encoding.UTF8.GetByteCount(builderS);
                    bytes += byteCount;
                    if (offset + byteCount > buffer.Length)
                    {
                        await stream.WriteAsync(buffer, 0, offset);
                        offset = 0;
                        if (byteCount > buffer.Length)
                        {
                            System.Diagnostics.Debug.WriteLine("Expanding buffer to {0} bytes", byteCount);
                            buffer = new byte[byteCount];
                        }
                    }
                    offset += System.Text.Encoding.UTF8.GetBytes(builderS, 0, builderS.Length, buffer, offset);
                }
            });
            await stream.WriteAsync(buffer, 0, offset);
            form = System.Text.Encoding.UTF8.GetBytes(boundary);
            await stream.WriteAsync(form, 0, form.Length);
            await stream.FlushAsync(); //pretty sure this does nothing
            System.Diagnostics.Debug.WriteLine("Wrote {0}.{1} megabytes of data", bytes / 1000000, bytes % 1000000);

I think the code above would work great if I were the server, just adding stream.Close(); would finish it, however since I am the client here closing it causes an error (TaskCancelled). Waiting to read doesn't do anything either, I presume because the PushStreamContent doesn't end the request unless I explicitly close the stream. That being said, writing to a file produces exactly what I expect to be uploaded so everything writes perfectly.

Any ideas on what I can do here? I might be totally misusing PushStreamContent but it seems like this should be an appropriate use case.

Answer 1

So the solution is a little confusing at first but it seems to make sense and perhaps more importantly, it works:

using(var content = new MultipartFormDataContent()) 
{
  var pushContent = new PushStreamContent(async (stream, httpContent, transportContext) =>
  { 
    //do the stream writing stuff
    stream.Close();
  });
  content.add(pushContent);
  //post, put, etc. content here
}

This works because the stream passed to the PushStreamContent method is not the actual request stream, it's a stream handled by the HttpClient, just like adding a file to a request stream. As a result, closing it signals the end of input for this part of the HttpContent and allows the request to be finalized.