CODEWITHSUNDEEP
c#multithreadingasync-awaitparallel.foreach
tonydev314
tonydev314

Reputation: 249

Simplest way to perform multiple long running processes concurrently c#

I have a long running job I need to run once for each item in a collection.

I'd like to do these jobs concurrently, although the entire program can wait for all jobs to complete. (I may change that later, but for now, I'm keeping the problem simple)

Following some help already given, I've got the following pattern:

public static void DoWork()
{
    //Get a collection of items
    var items = GetMyItems();

}


private async void DoStuffWithItems(ICollection<MyItem> items)
{
    var tasks = items.Select (i => DoStuffWithItem(i));
    await Task.WhenAll(tasks);
}

private Task DoStuffWithItem(MyItem item)
{
    //LongRunningTask
    return Task.Run(async () =>
            {
                var returnObject = await LongRunningAsyncMethod(item);
            });
}

If I understand it right though, THIS does each task one at a time still - so pointless.

Someone suggested I used Parallel.ForEach in combination with async await - the Parallel-ForEach pattern is easy enough:

public static void DoWork()
{
    //Get a collection of items
    var items = GetMyItems();
    
    Parallel.ForEach(items, (item) =>
    {
        var returnObject = LongRunningMethod(item);
    }

}

Now if I understand THIS correctly, I'll kick off a separate thread for each item to do its thing, but while it is doing this, the main thread will wait

But what if I want the rest of the program to carry on while DoWork is waiting for the Parallel.ForEach to complete?

public static void DoWork()
{

//Get a collection of items
var items = GetMyItems();

DoINeedThisMethod(items);


}

private async void DoINeedThisMethod(ICollection<MyItem> items)
    {
        await Task.Run(() => {
            DoStuffWithItems(items);
        });
    }

private async void DoStuffWithItems(ICollection<MyItem> items)
{
        var newobjs = new List<MyItem>();

            Parallel.ForEach(items, (item) =>
            {
                var bc = LongRunningAsyncMethod(item);
            });
}

It doesn't feel quite right - and I have an async void in there which isn't good practice. But am I along the right lines here? If I understand correctly, in DoINeedThisMethod, I'm spinning up a new thread (via Task) to perform the Parallel.ForEach call, but the use of "wait" in there means that the main thread will now continue while the Parallel.ForEach completes.

Upvotes: 0

Views: 2004

Answers (2)

Johnathan Barclay
Johnathan Barclay

Reputation: 20363

As confirmed by @Johan, the calls to DoStuffWithItem should happen simultaneously.

This is because the method returns Task, which may be clearer if you elide the lambda expression, which is completely unnecessary:

var tasks = items.Select(DoStuffWithItem);

Also, you shouldn't use Task.Run in the implementation of DoStuffWithItem. If LongRunningAsyncMethod is truly asynchronous, it will release the thread whilst I/O occurs anyway, which in turn allows your Select to generate the tasks in parallel.

Using Task.Run simply adds unnecessary overhead through thread switching and allocation.

You should abandon the idea of combining Parallel.ForEach with await; they serve different purposes.

Parallel utilises multiple threads and is suited to CPU-bound work, whereas async releases threads whilst asynchronous work (I/O) happens.

You should stick to your Select / WhenAll version, but without the Task.Run, as long as LongRunningAsyncMethod is implemented asynchronously.

Upvotes: 1

Johan Donne
Johan Donne

Reputation: 3285

Actually, your first assumption

If I understand it right though, THIS does each task one at a time still

is wrong as can easily be proven:

    private static async Task Main(string[] args)
    {

        async Task DoStuffWithItem(int i)
        {
            // long running task
            Console.WriteLine($"processing item {i} started");
            await Task.Delay(500);
            Console.WriteLine($"processing item {i} finished");
        }
        
        var items = new List<int> { 1, 2, 3 };

        var tasks = items.Select(i => DoStuffWithItem(i)).ToList();
        await Task.WhenAll(tasks);

        Console.WriteLine("\nFinished");
    }

yields the following output:

processing item 1 started
processing item 2 started
processing item 3 started
processing item 2 finished
processing item 1 finished
processing item 3 finished

Finished

So the tasks are executed concurrently...

Note: as explained in a comment below, the 'ToList()' causes the Linq Query to be executed immediately (which starts the tasks immediately). Without the 'ToList()', the tasks will only be started upon executing 'await Task.WhenAll(tasks)'

Upvotes: 5

Related Questions