CODEWITHSUNDEEP
c#asp.netasp.net-web-apiwebrequestparallel.foreach
DevDevDev
DevDevDev

Reputation: 649

Parallel.foreach() with WebRequest. 429 error

I am currently running a script that is hitting an api roughly 3000 times over an extended period of time. I am using parallel.foreach() so that I can send a few requests at a time to speed up the process. I am creating two requests in each foreach iteration so I should have no more than 10 requests. My question is that I keep receiving 429 errors from the server and I spoke to someone who manages the server and they said they are seeing requests in bursts of 40+. With my current understanding of my code, I don't believe this is even possible, Can someone let me know If I am missing something here?

    public static List<Requests> GetData(List<Requests> requests)
    {
        ParallelOptions para = new ParallelOptions();
        para.MaxDegreeOfParallelism = 5;

        Parallel.ForEach(requests, para, request =>
        {
            WeatherAPI.GetResponseForDay(request);
        });

        return requests;
    }

    public static Request GetResponseForDay(Request request)
    {
        var request = WebRequest.Create(request);
        request.Timeout = 3600000;
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        StreamReader myStreamReader = new 
        StreamReader(response.GetResponseStream());
        string responseData = myStreamReader.ReadToEnd();
        response.Close();

        var request2 WebRequest.Create(requestthesecond);
        HttpWebResponse response2 = (HttpWebResponse)request2.GetResponse();
        StreamReader myStreamReader2 = new 
        StreamReader(response2.GetResponseStream());
        string responseData2 = myStreamReader2.ReadToEnd();
        response2.Close();

        DoStuffWithData(responseData, responseData2)

        return request;
    }

Upvotes: 1

Views: 1735

Answers (2)

DevDevDev
DevDevDev

Reputation: 649

Both answers above are essentially correct. The problem I had was that the server manager set a rate limit of 100 requests/min where there previously wasn't one and didn't inform me. As soon as I hit this limit of 100 requests/min, I started receiving 429 errors nearly instantly and started firing off requests at that speed which were also receiving 429 errors.

Upvotes: 1

tlt
tlt

Reputation: 15211

as smartobelix pointed out, your degreeofparallelism set to 5 doesn't prevent you of sending too many requests as defined by servers side policy. all it does is prevents you from exhausting number of threads needed for making those requests on your side. so what you need to do is communicate to servers owner, get familiar with their limits and change your code to never meet the limits.

variables involved are:

  1. number of concurrent requests you will send (parallelism)
  2. average time one request takes
  3. maximum requests per unit time allowed by server

so, for example, if your average request time is 200ms and you have max degree of parallelism of 5, then you can expect to send 25 requests per second on average. if it takes 500ms per request, then you'll send only 10.

mix servers allowed numbers into that and you'll get the idea how to fine tune your numbers.

Upvotes: 1

Related Questions