Streaming Json - PushStreamContent with a Large Object

Question

I need to stream a large object. I'm unable to figure out how to send it in chunks. The posted code works, however, stream.Flush() only gets called once. So, essentially I'm buffering the object - not good. How do I call stream.Flush() multiple times? If I had a collection, I could stream/flush in a loop. So how do I do that with a large object?

SERVER CODE:

   public async Task<HttpResponseMessage> ConvertToTiffAsync([FromBody] DocumentDto dto)
                {
                    // THIS IS LARGE
                    var document = await _service.ConvertToTiffAsync(dto);
                    var response = Request.CreateResponse();
                    response.Content = new PushStreamContent((stream, content, context) =>
                    {
                        var serializer = new JsonSerializer();
                        using (var writer = new StreamWriter(stream))
                        {
                            using (var jsonTextWriter = new JsonTextWriter(writer))
                            {
                                serializer.Serialize(jsonTextWriter, document);
                                stream.Flush(); // ONLY CALLED ONCE - NEED MANY CALLS
                            }
                        }
                    });
                    return response;
                }

CLIENT CODE (streaming not happening here, but needs to):

using (var client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true }))
            {
                client.Timeout = new TimeSpan(0, 5, 0);
                var stringContent = new StringContent(JsonConvert.SerializeObject(dto), Encoding.UTF8, "application/json");
                using (var httpRequest = new HttpRequestMessage(HttpMethod.Post, endpoint))
                {
                    httpRequest.Content = stringContent;
                    using (HttpResponseMessage response = await client.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false))
                    {
                        response.EnsureSuccessStatusCode();
                        using (var streamReader = new StreamReader(await response.Content.ReadAsStreamAsync()))
                        using (var jsonReader = new JsonTextReader(streamReader))
                        {
                            var serializer = new JsonSerializer();
                            return await Task.Run(() => serializer.Deserialize<ConvertDocumentDto>(jsonReader)).ConfigureAwait(false);
                        }
                    }
                }
            }

Answer 1

Your server code seems to be fine. However, your client code doesn't seem to be able to properly interact with the server.

If you want to set DocumentDto input, you would need to write the JSON to the request, not to the response. If you want to consume the server action without having the complete JSON in memory you would need a JsonTextReader. So in general in the client-side should be:

var client = new HttpClient();
var dtoContent = new PushStreamContent((stream, content, context) => {
   var serializer = new JsonSerializer();
    using (var streamWriter = new StreamWriter(stream))
    {
       using (var jsonWriter = new JsonTextWriter(streamWriter))
       { serializer.Serialize(jsonTextWriter, dto); }
    }
});
using (var stream = await client.PostAsync(url, dtoContent).Content.ReadAsStreamAsync())
{
  using (var streamReader = new StreamReader(stream))
  {
    using (var jsonReader = new JsonTextReader(streamReader))
    {
        document = JsonSerializer().Deserialize<Document>(jsonReader);
    }
  }
}

If your DTO is small and you don't mind having the complete JSON in memory instead of client.PostAsync(url) you could also use client.PostAsJsonAsync(url, dto) which is in the HttpClientExtensions namespace if you import System.Net.Http.Formatting.dll.