CODEWITHSUNDEEP
c#asynchronousforeachasync-await
Jacob
Jacob

Reputation: 689

Multiple async/await calls in a foreach loop iteration

I am trying to wrap my head around how to handle multiple async/await calls in a foreach loop. I have around 20,000 rows of data that are processed by the foreach loop. Roughly my code is:

foreach (var item in data)
{
    if (ConditionA(item))
    {
        if (ConditionAB(item));
        {
            await CreateThingViaAPICall(item)
        }
        else
        {
            var result = await GetExistingRecord(item);
            var result2 = await GetOtherExistingRecord(result);
            var result3 = await GetOtherExistingRecord(result2);
            //Do processing
            ...           
            await CreateThingViaAPICall();
        }
    }
    ... and so on        
}

I've seen many posts saying the best way to use async in a loop is to build a list of tasks and then use Task.WhenAll. In my case I have Tasks that depend on each other as part of each iteration. How do I build up a list of tasks to execute in this case?

Upvotes: 3

Views: 4377

Answers (2)

Todd Menier
Todd Menier

Reputation: 39309

It's easiest if you break the processing of an individual item into a separate (async) method:

private async Task ProcessItemAsync(Item item)
{
    if (ConditionA(item))
    {
        if (ConditionAB(item));
        {
            await CreateThingViaAPICall(item)
        }
        else
        {
            var result = await GetExistingRecord(item);
            var result2 = await GetOtherExistingRecord(result);
            var result3 = await GetOtherExistingRecord(result2);
            //Do processing
            ...           
            await CreateThingViaAPICall();
        }
    }
    ... and so on
}

Then process your collection like so:

var tasks = data.Select(ProcessItemAsync);
await Task.WhenAll(tasks);

This effectively wraps the multiple dependent Tasks required to process a single item into one Task, allowing those steps to happen sequentially while items of the collection itself are processed concurrently.

With 10's of thousands of items, you may, for a variety of reasons, find that you need to throttle the number of Tasks running concurrently. Have a look at TPL Dataflow for this type of scenario. See here for an example.

Upvotes: 3

Jon Hanna
Jon Hanna

Reputation: 113242

If I'm not mistaken the recommended way to use async/wait in a foreach is to build a list of Tasks first then call Task.WhenAll.

You're partly mistaken.

If you have a multiple tasks that don't depend on each other then it is indeed generally a very good idea to have those multiple task happen in a WhenAll so that they can be scheduled together, giving better throughput.

If however each task depends on the results of the previous, then this approach isn't viable. Instead you should just await them within a foreach.

Indeed, this will work fine for any case, it's just suboptimal to have tasks wait on each other if they don't have to.

The ability to await tasks in a foreach is in fact one of the biggest gains that async/await has given us. Most code that uses await can be re-written to use ContinueWith quite easily, if less elegantly, but loops were trickier and if the actual end of the loop was only found by examining the results of the tasks themselves, trickier again.

Upvotes: 1

Related Questions