Running out of sockets when re-using a single HttpClient amongst multiple threads - 1000's of TIMED_WAIT connections remain

Question

Environment: Windows Server 2012 R2 64-bit. C# .NET Framework version 4.5.1.

I am trying to use this program to download a bunch of files from a SharePoint 2013 site: https://github.com/nddipiazza/Sharepoint-Exporter

In this project there is a [FileDownloader.cs][1] file that pulls requests to download files from a BlockingCollection and downloads them to file.

When I run this I pretty quickly get hit with socket errors:

System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 10.5.50.2:443
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at SpPrefetchIndexBuilder.FileDownloader.attemptToDownload(FileToDownload toDownload, Int32 numRetry)
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted 10.5.50.2:443
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---<---

Basically it seems like I am hitting this code too hard:

public void AttemptToDownload(FileToDownload toDownload, int numRetry)
{
    try
    {
        var responseResult = client.GetAsync(SpPrefetchIndexBuilder.topParentSite + toDownload.serverRelativeUrl);
        if (responseResult.Result != null && responseResult.Result.StatusCode == System.Net.HttpStatusCode.OK)
        {
            using (var memStream = responseResult.Result.Content.ReadAsStreamAsync().GetAwaiter().GetResult())
            {
                using (var fileStream = File.Create(toDownload.saveToPath))
                {
                    memStream.CopyTo(fileStream);
                }
            }
            Console.WriteLine("Thread {0} - Successfully downloaded {1} to {2}", Thread.CurrentThread.ManagedThreadId, toDownload.serverRelativeUrl, toDownload.saveToPath);
        }
        else
        {
            Console.WriteLine("Got non-OK status {0} when trying to download url {1}", responseResult.Result.StatusCode, SpPrefetchIndexBuilder.topParentSite + toDownload.serverRelativeUrl);
        }
    }
    catch (Exception e)
    {
        if (numRetry >= NUM_RETRIES)
        {
            Console.WriteLine("Gave up trying to download url {0} to file {1} after {2} retries due to error: {3}", SpPrefetchIndexBuilder.topParentSite + toDownload.serverRelativeUrl, toDownload.saveToPath, NUM_RETRIES, e);
        }
        else
        {
            AttemptToDownload(toDownload, numRetry + 1);
        }
    }
}

Is there something wrong with how I'm using HttpClient? I read all the forums and it says to re-use a static reference, and I am doing that. All of my threads share the same static HttpClient reference.

Here is a netstat -a -o -n from when the downloads have been running for a few minutes. There are a lot of TIMED_WAIT sitting there. https://pastebin.com/GTYmqwue In this test i was using SharePoint on port 80. Why is that?

When I run it again a couple minutes later, the number of TIMED_WAIT have increased hundreds more. There must be a leak of some sort going on?

Why is HttpClient leaving 1000's of TIMED_WAIT connections sitting there? How can I get them to close?

I tried to set ServiePointManager.DefaultConnectionLimit = numThreads but it still grows to 1000's of connections.

Answer 1

I think I figured it out finally

I accidentally had a really aggressive client.Timeout = TimeSpan.FromMinutes(5);

I changed this to client.Timeout = TimeSpan.FromSeconds(15);

Now I think the connections aren't leaking anymore.