CODEWITHSUNDEEP
c#.netmultithreadinghttpwebrequest
John Smith
John Smith

Reputation: 23

Threading HTTPWebRequests

I have a program which is permanently sending http requests to a server and analyzing the responses. As I need to send a lot of requests I would like to do parallel http requests. I tried a few things like Parallel.ForEach, creating Tasks with the longrunningoption, etc. but I always had the same problem, it was too slow. I'm using the httpwebrequest class and for testing I compared the times which one thread needed for doing 50 requests, and then the same with more threads. 1 Thread took 1500ms for these 50 requests, and 5 threads already took 1900-2500ms. It slowed down with each new thread/task..

    public void test()
    {
        Stopwatch stopWatch = new Stopwatch();

        stopWatch.Start();

        for (int i = 0; i < 50; i++)
        {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("www.thetestingurl.com");

        request.Method = "GET";

        var response = request.GetResponse();

        response.Close();            
        }

        stopWatch.Stop();

        MessageBox.Show(stopWatch.ElapsedMilliseconds.ToString());
    }

So what am I doing wrong? I already know about ServicePointManager.DefaultConnectionLimit, but this doesnt help.

EDIT:

This is how I try to create the tasks:

for (int i = 0; i < taskCount; i++)
{
    var task = new Task(() => test(),
        TaskCreationOptions.LongRunning);
    task.Start();
}
Task.WaitAll();

EDIT:

So, to make it clear what I mean: When I run this code on 1 Thread it takes 1500ms, when I use tasks, threads, threadpool etc. each thread needs more than 1500ms. I want to create x Threads, they should send the requests parallel and each thread should take 1500ms for the 50 requests, just as the 1 threaded application does. Is this not possible?

Upvotes: 2

Views: 5506

Answers (1)

nikolai.serdiuk
nikolai.serdiuk

Reputation: 762

I suggest using ThreadPool instead of Task or Parallel.ForEach, because they also uses ThreadPool and you have no control on it. I prepared simple example based on your code, you can compile it as console app:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;

namespace Test1
{
    public class Class1
    {
        public void test()
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.dir.bg/");

            request.Method = "GET";

            var response = request.GetResponse();

            response.Close();
        }

        public static void Main()
        {
            CountdownEvent countDown = new CountdownEvent(50);

            Stopwatch sw = new Stopwatch();
            sw.Start();


            for (int i = 0; i < 50; i++)
            {
                //11646 ms
                //new Class1().test();

                // 502 ms
                ThreadPool.QueueUserWorkItem(DoRequest, countDown);
            }

            countDown.Wait();

            sw.Stop();

            MessageBox.Show(sw.ElapsedMilliseconds.ToString());


        }

        private static void DoRequest(Object stateInfo)
        {
            CountdownEvent countDown = (CountdownEvent)stateInfo;

            new Class1().test();

            countDown.Signal();
        }
    }


}

In my tests time was decreased from 11646 ms to 502 ms.

Upvotes: 2

Related Questions