CODEWITHSUNDEEP
c#httpwebrequest
loveforfire33
loveforfire33

Reputation: 1120

HttpWebRequest Slows with multiple instances of application

trying to get to the bottom of this!

i have a very basic app that is using httpwebrequests to login, navigate to a page and then grab the html of that page. it then preforms another webrequest to a third page every 5 mins in a loop.

its all working fine and is single threaded (and fairly old), however circumstances have changed and i now need to run multiple instances of this app closely together (i have a .bat starting the app every 2seconds as a temporary measure until i am able to code a new multithreaded solution).

when the first instances of the app start everything is fine, first request is completed in ~2seconds. second one in about 3seconds.

however as more and more instances of this app are run concurrently (>100) something strange starts to happen.

the first web request still takes ~2 seconds, however the second request gets delayed much more >1min up to the point of timeout. i cant seem to think why this is. the second page is larger than the first, but nothing out of the ordinary that would take >1min to download.

The internet connection and hardware of this server is more than capable of handling these requests.

 CookieContainer myContainer = new CookieContainer();
        // first request is https
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(https://mysite.com/urlone);                                
        request.CookieContainer = myContainer;
        request.Proxy = proxy;
        Console.WriteLine(System.DateTime.Now.ToLongTimeString() + "  " + "Starting login request");
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        Stream resStream = response.GetResponseStream();

        string tempString = null;
        int count = 0;

        do
        {
            // fill the buffer with data
            count = resStream.Read(buf, 0, buf.Length);

            // make sure we read some data
            if (count != 0)
            {
                // translate from bytes to ASCII text
                tempString = Encoding.ASCII.GetString(buf, 0, count);

                // continue building the string
                sb.Append(tempString);
            }
        }
        while (count > 0); // any more data to read?
        sb.Clear();
       response.Close();
       resStream.Close();
       string output6;



        Console.WriteLine(System.DateTime.Now.ToLongTimeString() + "  " + "login request comeplete");
        HttpWebRequest request6 = (HttpWebRequest)WebRequest.Create(@"http://mysite.com/page2");
        request6.CookieContainer = myContainer;
        response = (HttpWebResponse)request6.GetResponse();
        resStream = response.GetResponseStream();
        tempString = null;
        count = 0;

        do
        {
            count = resStream.Read(buf, 0, buf.Length);


            if (count != 0)
            {
                tempString = Encoding.ASCII.GetString(buf, 0, count);

                sb.Append(tempString);
            }
        }

        while (count > 0);
        output6 = sb.ToString();
        sb.Clear();
        response.Close();
        resStream.Close();

Any ideas? Im not very advanced with http web requests so if someone could check i haven't made any silly code mistakes above id appreciate it. Im at a loss as to what other information i may need to include here, if i have missed anything out please tell me and i will do my best to provide. Thanks in advance.

EDIT 1:

I used fiddler to find out the source of the issue. It looks like the issue lies with the application (or windows) not sending the requests for some reason - the physical request actually takes < 1second according to fiddler.

Upvotes: 0

Views: 441

Answers (2)

Nicolas78
Nicolas78

Reputation: 5144

Try wrapping your request into a using directive to make sure it's always properly closed. If you hit the max number of connections, you otherwise have to wait for the earlier ones to time out before new ones connect.

Upvotes: 0

Parimal Raj
Parimal Raj

Reputation: 20585

Check out a few things

  1. ServicePointManager.DefaultConnectionLimit : if you are planning to open more then 100 connection the set this value to something like 200-300.
  2. If possible use HttpWebRequest.KeepALive = true

Upvotes: 2

Related Questions