CODEWITHSUNDEEP
c#asynchronouswhile-loopasync-await
PinoyDev
PinoyDev

Reputation: 1047

how to loop through rest api without too many requests using httpClient

I'm kinda new to async and await. Whenever I call API repeatedly I get a too many request error. So my question is who do I loop through an API call using Async and Await so I don't get penalized? See following code example.

Task <IDList> ListOfIDs = new Task(GetIDListAsycn);
var result  = ListOfIDs.Wait()

Once I get the results, I want to loop through them and get the ID to pass to the API URL like this until I ran out of IDs:

foreach(var n in List) { client.BaseUrl (new Uri("http://api/users/n.ID"))};

Problem is like I said whenever I attempt this kind of logic I get too many request because the client.GetStringAsync is called so many times in short period of time.

Upvotes: 2

Views: 5588

Answers (2)

Babak
Babak

Reputation: 1344

As @JuanR mentioned, the issue rises where too many requests are sent to the API faster than they could be handled.

To overcome this issue and have more control over sending requests, I can think of two approaches:

Using an async Method

You can define GetItemsByID as async and wait for the response to each request before proceeding to send the next one.

private static async Task GetItemsAsync(List<dynamic> list)
{
    var client = new HttpClient();

    foreach (var n in list)
    {
        var res = await client.GetAsync("http://api/users/" + n.ID);

        // Do whatever you want with `res` here
    }
}

Calling the async method

  • Call it inside an async method like this:
await GetItemsAsync(List);
  • Call it inside a non-async method:
GetItemsAsync(List).Wait();

More Control using ActionBlock

You can use ActionBlock so that multiple calls to the API at the same time "could" make the whole process faster. It is possible to limit the number of parallel calls using MaxDegreeOfParallelism:

private static void GetItemsByID(List<dynamic> list)
{
    var client = new HttpClient();
    var workerBlock = new ActionBlock<string>(async id =>
        {
            var res = await client.GetAsync("http://api/users/" + id);

            // Do whatever you want with `res` here
        },
        new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 4 }
    );

    foreach (var n in list)
    {
        workerBlock.Post(n.ID);
    }
    workerBlock.Complete();

    // Wait for all messages to propagate through the network.
    workerBlock.Completion.Wait();
}

Calling the method

  • Call it simply like any other method:
GetItemsById(List);

Read More

Upvotes: 4

JuanR
JuanR

Reputation: 7783

Your issue is at the loop level. The loop is executing too fast, sending a lot of calls to the API, which causes the server to complain about too many requests.

You need to throttle it by slowing down the calls:

foreach(var n in List) 
{ 
    client.BaseUrl(new Uri("http://api/users/" + n.ID));

    //Add this to make the thread sleep for a second.
    System.Threading.Thread.Sleep(1000);
}

Change the milliseconds to whatever works for you.

Upvotes: 2

Related Questions