Multiple simultaneous WebClient http calls are slow to finish

Question

I'm having a startup problem in a web service hosted in the IIS. The service needs to fetch external resources over http online to be used when servicing requests. To simplify this these resources are collected in background threads and cached. And since I don't want the initial requests to the service to fail these resources are all collected at once during startup. They're downloaded using the System.Net.WebClient class, and what happens is that a bunch of threads are created who all have one WebClient object which they use to download a resource.

And this is behaving strangely. It's almost as if multiple requests are blocking eachother somehow, because what happens is that all of the requests needs ages to finish. While all of these requests are running I can call a web site hosted locally that consists of a small "hello world" http-page and it will take seconds and seconds (with my current setup it takes 0.1 seconds if called before the background thread is started and 30 if called after it's started). After about a minute things go back to normal and instead of not being able to reach any of the resources it can suddenly download most of them in under a second.

Experimentation shows that this is not a problem with limits on concurrent threads or connections (there aren't any successful calls at all for a minute - if there was some kind of connection limit you'd expect a few calls to succeed). Another interesting experiment was when I instead tried to connect to the small local site using pure sockets - this worked flawlessly, so it seems to be a problem specifically with the WebClient class.

The classes used to make the connection are found below. Calls to the TimeoutRequestHandler are pretty straightforward - you create it using an uri and a CustomWebClient and you call process request.

    private class CustomWebClient : WebClient {

        public bool KeepAlive { get; set; }

        public X509Certificate ClientCertificate { get; set; }

        protected override WebRequest GetWebRequest(Uri address) {

            WebRequest answer = base.GetWebRequest(address);

            HttpWebRequest httpReq = answer as HttpWebRequest;
            if (httpReq != null) {
                httpReq.KeepAlive = KeepAlive;

                if (ClientCertificate != null) {
                    httpReq.ClientCertificates.Add(ClientCertificate);
                }
            }
            return answer;
        }
    }

    private class TimeoutRequestHandler {

        private readonly Uri address;
        private readonly WebClient client;
        private readonly byte[] requestData;
        private readonly TimeSpan timeout;

        private readonly object sync;
        private ManualResetEvent requestDoneSignal;
        private AsyncCompletedEventArgs completedInfo;

        public TimeoutRequestHandler(Uri address, WebClient client, byte[] requestData, TimeSpan timeout) {

            this.address = address;
            this.client = client;
            this.requestData = requestData;
            this.timeout = timeout;

            sync = new object();
            requestDoneSignal = new ManualResetEvent(false);
            client.UploadDataCompleted += OnRequestCompleted;
            client.DownloadDataCompleted += OnRequestCompleted;
        }

        public byte[] ProcessRequest() {

            bool shouldCancel = false;
            try {
                if (requestData != null) {
                    client.UploadDataAsync(address, requestData); // Uses POST for HTTP.
                } else {
                    client.DownloadDataAsync(address); // Uses GET for HTTP
                }
                if (!requestDoneSignal.WaitOne(timeout)) {
                    shouldCancel = true;
                    throw new WebException("The operation has timed out");
                }
                if (completedInfo.Cancelled) {
                    throw new WebException("The operation has been cancelled");
                }
                if (completedInfo.Error != null) {
                    throw completedInfo.Error;
                }
                return GetResponseData(completedInfo);
            } finally {
                AllDone(shouldCancel);
            }
        }

        private byte[] GetResponseData(AsyncCompletedEventArgs e) {

            return e is UploadDataCompletedEventArgs ?
                ((UploadDataCompletedEventArgs)e).Result :
                ((DownloadDataCompletedEventArgs)e).Result;
        }

        private void OnRequestCompleted(object sender, AsyncCompletedEventArgs e) {

            lock (sync) {
                if (requestDoneSignal != null) {
                    completedInfo = e;
                    requestDoneSignal.Set();
                }
            }
        }

        private void AllDone(bool shouldCancel) {

            lock (sync) {
                requestDoneSignal.Close();
                requestDoneSignal = null;
                client.UploadDataCompleted -= OnRequestCompleted;
                client.DownloadDataCompleted -= OnRequestCompleted;
                if (shouldCancel) {
                    client.CancelAsync();
                }
            }
        }
    }
}

Answer 1

Not sure if it's related but I've seen issues where resolving the proxy server can be very slow and add a lot of overhead. I recently added the following lines to a project to improve a WebClient performance:

ServicePointManager.DefaultConnectionLimit = 256;
WebRequest.DefaultWebProxy = null;

ServicePointManager.DefaultConnectionLimit Property:

The DefaultConnectionLimit property sets the default maximum number of concurrent connections that the ServicePointManager object assigns to the ConnectionLimit property when creating ServicePoint objects.

The documentation for ServicePointManager.DefaultConnectionLimit is not clear as it states:

The default value is Int32.MaxValue.

But then goes on to state:

When used in the server environment (ASP.NET) DefaultConnectionLimit defaults to higher number of connections, which is 10.

I just watched the value in a .NET 4.5 WPF application in Visual Studio 2015 and it defaulted to 2:

WebRequest.DefaultWebProxy Property:

The DefaultWebProxy property gets or sets the global proxy. The DefaultWebProxy property determines the default proxy that all WebRequest instances use if the request supports proxies and no proxy is set explicitly using the Proxy property. Proxies are currently supported by FtpWebRequest and HttpWebRequest.

The DefaultWebProxy property reads proxy settings from the app.config file. If there is no config file, the current user's Internet Explorer (IE) proxy settings are used.

If the DefaultWebProxy property is set to null, all subsequent instances of the WebRequest class created by the Create or CreateDefault methods do not have a proxy.

The following config parameters are also available:

<system.net>
  <connectionManagement>
    <add address="*" maxconnection="256"/>
  </connectionManagement>
  <defaultProxy enabled="false" />
</system.net>

<connectionManagement> Element (Network Settings)

<defaultProxy> Element (Network Settings)

Obviously setting enabled="false" on defaultProxy won't work if you have a proxy. In that case I would specify the proxy details in the config so it doesn't have to check IE.