CODEWITHSUNDEEP
c#for-loopasync-await
Ian Vink
Ian Vink

Reputation: 68740

Running groups of groups of Tasks in a For loop

I have a set of 100 Tasks that need to run, in any order. Putting them all into a Task.WhenAll() tends to overload the back end, which I do not control.

I'd like to run n-number tasks at a time, after each completes, then run the next set. I wrote this code, but the "Console(Running..." is printed to the screen all after the tasks are run making me think all the Tasks are being run.

How can I force the system to really "wait" for each group of Tasks?

//Run some X at a time
int howManytoRunAtATimeSoWeDontOverload = 4;
for(int i = 0; i < tasks.Count; i++)
{
    var startIndex = howManytoRunAtATimeSoWeDontOverload * i;
    Console.WriteLine($"Running {startIndex} to {startIndex+ howManytoRunAtATimeSoWeDontOverload}");

    var toDo = tasks.Skip(startIndex).Take(howManytoRunAtATimeSoWeDontOverload).ToArray();
    if (toDo.Length == 0) break;
    await Task.WhenAll(toDo);
}

Screen Output:

Screen Output

Upvotes: 0

Views: 97

Answers (1)

vdL
vdL

Reputation: 277

There are a lot of ways to do this but I would probably use some library or framework that provides a higher level abstraction like TPL Dataflow: https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/dataflow-task-parallel-library (if your using .NET Core there's a newer library).

This makes it a lot easier than building your own buffering mechanisms. Below is a very simple example but you can configure it differently and do a lot more with this library. In the example below I don't batch them but I make sure no more than 10 tasks are processed at the same time.

        var buffer = new ActionBlock<Task>(async t =>
        {
            await t;
        }, new ExecutionDataflowBlockOptions { BoundedCapacity = 10, MaxDegreeOfParallelism = 1 });

        foreach (var t in tasks)
        {
            await buffer.SendAsync(DummyFunctionAsync(t));
        }

        buffer.Complete();
        await buffer.Completion;

Upvotes: 2

Related Questions