"Unable to read data from the transport connection: An existing connection was (NOT FORCIBLY) closed" on a RESTful Web Service call

Question

I have read numerous Stack Overflow questions (such as this one) discussing variants of the following error message:

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

However, I'm getting the following variant on the error message:

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was closed.

No indication of "forcibly closed by the remote host." There were no further details in the exception message as to exactly why it was closed (or even whether it was the client or server who closed the connection).

Why could this happen? Does this mean that there was a timeout on my end or something, or is this a problem on our vendor's side? (I'd like to be sure whose "fault" the problem is before I raise a ticket with our vendor to investigate).

The particular file I'm downloading is quite large and could potentially take a long time to download.

My code is pretty "standard" HttpClient "get" logic:

    private async Task<string> GetReport(string fileNamePrefix, bool lookInCurrentFolder = false, bool returnEntireThing = true)
    {
        string fileName = $"{fileNamePrefix}-report-{DateTime.Now.ToString("yyyy-MM-dd")}.csv";

        string downloadFolder = FileUtilities.GetPath(KnownFolder.Downloads);

        string fileInDownloadFolder = Path.Combine(downloadFolder, fileName);

        if ((lookInCurrentFolder && !File.Exists(fileName)) || (!lookInCurrentFolder && !File.Exists(fileInDownloadFolder)))
        {
            HttpClient client = HttpClientConstructor.GetHttpClient(acceptVersion: false, acceptType: "text/csv");
            client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));

            using (HttpResponseMessage msg = await client.GetAsync("analytics/" + fileNamePrefix))
            {
                using (var fstr = new FileStream(lookInCurrentFolder ? fileName : fileInDownloadFolder, FileMode.Create))
                {
                    using (GZipStream str = new GZipStream(await msg.Content.ReadAsStreamAsync(), CompressionMode.Decompress))
                    {
                        str.CopyTo(fstr);
                    }
                }
            }
        }

        return returnEntireThing ? File.ReadAllText(lookInCurrentFolder ? fileName : fileInDownloadFolder) : (lookInCurrentFolder ? fileName : fileInDownloadFolder);
    }

Abbreviating slightly, the HttpClientConstructor is a factory that returns an appropriate HttpClient Singleton (since Microsoft recommends using one HttpClient throughout the application lifecycle). It's possible that there will be several instances of HttpClient with several different acceptType configurations. The instances are created more or less like this:

           var client = new HttpClient
            {
                BaseAddress = new Uri("..."),
                Timeout = new TimeSpan(4, 0, 0)
            };

            client.DefaultRequestHeaders.Add("ApiToken", key);
            client.DefaultRequestHeaders.Accept.Clear();
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(acceptType));

What's going wrong here? Why the weird error message? Is this likely to be a problem on my end?

Edit: I am setting the security settings like this before I create my HttpClient instances in HttpClientConstructor:

ServicePointManager.DefaultConnectionLimit = 8;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

Also, this problem is intermittent - the code works most of the time, this is the first time I've seen this happen.

Answer 1

I had a similar problem with the use of a shared HttpClient connecting to a server for REST calls. The problem ended up being a mismatch between the KeepAlive timeout on the client and server. The client side timeout is set by the MaxServicePointIdleTime setting on the ServicePointManager and defaults to 100s. The server side idle timeout was set to a shorter value in our server.

Having a shorter timeout on the server as compared to the client resulted in the server sporadically closing a connection just when the client was attempting to connect. This resulted in the reported exception.

Note that I ultimately found the problem because I also received this exception under the same conditions:

System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

Although the problem could be fixed by using a new HttpClient for each connection to the server, this goes against Microsoft's recommendation for sharing the HttpClient. I'd suggest putting the MaxServerPointIdleTime to a shorter value, perhaps 15 seconds, to see if that addresses the issue. Alternately, if you have access to the server code, you could find its timeout, and attempt to set it to a value longer then the 100s timeout defaulted to on the client.