Parallel Mulit-threaded Downloads using async-await

Question

I have 100s of multiple big files to download from web in my windows service - C#. The requirement is to maintain at one time - max 4 parallel web file downloads.

Can I achieve concurrent/parallel downloads using async await or do I have to use BackgroundWorker process or threads ? Is async-await multithreaded ? See my sample Program using async-await below:

 static int i = 0;
 
 Timer_tick() {
   while (i < 4) {
     i++;
     model = GetNextModel();
     await Download(model);
   }
 }
 
 private async Download(XYZ model) {
   Task<FilesetResult> t = DoWork(model);
   result = await t;
   //Use Result
 }
 
 private async Task<FilesetResult> Work(XYZ model) {
   fileresult = await api.Download(model.path)
   i--;
   return filesetresult;
 }

Answer 1

Doing it by hand seems awfully complicated.

var files = new List<Uri>();

Parallel.ForEach(files, 
                 new ParallelOptions { MaxDegreeOfParallelism = 4 },
                 this.Download);

Now all you need is a single, normal, synchronous method private void Download(Uri file) and you are good to go.

If you need a producer/consumer pattern, the easiest version might be a BlockingCollection:

using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApp11
{
    internal class Program
    {
        internal static void Main()
        {
            using (var queue = new BlockingCollection<Uri>())
            {
                // starting the producer task:
                Task.Factory.StartNew(() =>
                {
                    for (int i = 0; i < 100; i++)
                    {
                        // faking read from message queue... we get a new Uri every 100 ms
                        queue.Add(new Uri("http://www.example.com/" + i));

                        Thread.Sleep(100);
                    }

                    // just to end this program... you don't need to end this, just listen to your message queue
                    queue.CompleteAdding();
                });

                // run the consumers:
                Parallel.ForEach(queue.GetConsumingEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = 4 }, Download);
            }
        }

        internal static void Download(Uri uri)
        {
            // download your file here

            Console.WriteLine($"Downloading {uri} [..        ]");
            Thread.Sleep(1000);
            Console.WriteLine($"Downloading {uri} [.....     ]");
            Thread.Sleep(1000);
            Console.WriteLine($"Downloading {uri} [.......   ]");
            Thread.Sleep(1000);
            Console.WriteLine($"Downloading {uri} [......... ]");
            Thread.Sleep(1000);
            Console.WriteLine($"Downloading {uri} [..........]");
            Thread.Sleep(1000);
            Console.WriteLine($"Downloading {uri} OK");
        }
    }
}

Answer 2

You can limit number of async tasks running in parallel using SemaphoreSlim class. Something like:

List<DownloadRequest> requests = Enumerable.Range(0, 100).Select(x => new DownloadRequest()).ToList();
using (var throttler = new SemaphoreSlim(4))
{
    Task<DownloadResult>[] downloadTasks = requests.Select(request => Task.Run(async () =>
    {
        await throttler.WaitAsync();
        try
        {
            return await DownloadTaskAsync(request);
        }
        finally
        {
            throttler.Release();
        }
    })).ToArray();
    await Task.WhenAll(downloadTasks);
}

Update: thank you for comments, fixed issues.

Update2: Sample solution for dynamic list of requests

public class DownloadManager : IDisposable
{
    private readonly SemaphoreSlim _throttler = new SemaphoreSlim(4);

    public async Task<DownloadResult> DownloadAsync(DownloadRequest request)
    {
        await _throttler.WaitAsync();
        try
        {
            return await api.Download(request);
        }
        finally
        {
            _throttler.Release();
        }
    }

    public void Dispose()
    {
        _throttler?.Dispose();
    }
}