Waiting for all async downloads to complete

Question

I'm trying to run some code that downloads multiple files, and waiting for all to be completed before continuing with the code execution.

I currently have this part of code running, which actually does download everything as it's supposed to, but it won't resume the execution.. The program just freezes.

Would be great if you could help me with resolving this, thanks.

private void DownloadFiles(string[] targets, string sub_id, string base_url)
{
    var tasks = new List<Task>();
    int c = 0;

    foreach (var target in targets)
    {
        if (target.EndsWith(".vtt", StringComparison.InvariantCultureIgnoreCase))
        {
            using (var wc = new WebClient())
            {
                var task = DownloadFile(wc, base_url + sub_id + "/" + target, sub_id + "." + c++ + ".vtt");
                tasks.Add(task);
            }
        }
    }
    Task.WaitAll(tasks.ToArray());
}


private Task DownloadFile(WebClient wc, string target, string name)
{
    wc.DownloadProgressChanged += (object sender, DownloadProgressChangedEventArgs e) =>
    {
        Console.WriteLine(e.ProgressPercentage + "% downloaded.");
    };

    wc.DownloadFileCompleted += (object sender, AsyncCompletedEventArgs e) =>
    {
        Console.WriteLine(target + " was downloaded.");
    };

    return wc.DownloadFileTaskAsync(target, Environment.CurrentDirectory + "/Subs/" + name);
}

Answer 1

You should consider using Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:

var query =
    from x in targets.ToObservable().Select((t, c) => new { t, c })
    where x.t.EndsWith(".vtt", StringComparison.InvariantCultureIgnoreCase)
    let target = $"{base_url}{sub_id}/{x.t}"
    let name = $"{sub_id}.{x.c}.vtt"
    from status in
        Observable
            .Using(
                () => new WebClient(),
                wc =>
                {
                    var progress =
                        Observable
                            .FromEventPattern<DownloadProgressChangedEventHandler, DownloadProgressChangedEventArgs>(
                                h => wc.DownloadProgressChanged += h, h => wc.DownloadProgressChanged -= h)
                            .Select(ep => $"{ep.EventArgs.ProgressPercentage}% downloaded.");

                    var completed =
                        Observable
                            .FromAsync(() => wc.DownloadFileTaskAsync(target, $"{Environment.CurrentDirectory}/Subs/{name}"))
                            .Select(z => $"{target} was downloaded.");

                    return progress.Merge(completed);
                })
    select new { target, status };

That's one asynchronous query that handles all of the parallel calls - disposing the WebClient as it finishes each call.

You can get it to wait for all of the results like this:

query
    .Do(x => Console.WriteLine(x.status))
    .ToArray()
    .Wait();

But the more idiomatic way to handle it is this:

IDisposable subscription =
    query
        .Subscribe(
            x => Console.WriteLine(x.status),
            ex => { /* handle an exception */ },
            () => { /* when everything is done */ });

This processes the results as soon as they are available and gives you a chance to run some code when you're done.

If you need to marshall to a UI thread then you can do this:

IDisposable subscription =
    query
        .ObserveOnDispatcher() // or .ObserveOn(instanceOfForm)
        .Subscribe(
            x => Console.WriteLine(x.status),
            ex => { /* handle an exception */ },
            () => { /* when everything is done */ });

To stop the download in case you need to stop early just do subscription.Dispose();.

Answer 2

tl;dr

You'll want to await tasks and async methods

await Task.WhenAll(tasks.ToArray());
// ...
return await wc.DownloadFileTaskAsync(...);

This assumes a method with an async signature.

ok but why?

Setting aside the possibility that these void methods are event handlers --running on a UI thread in winforms, webforms, wpf, whatever-- your WaitAll is a blocking method. By waiting for all of those you are blocking the current thread. By awaiting all of those, you're allowing them to be run asynchronously.

If these void methods are also being run on the UI thread then that's a second, and similar, problem.

a longer example

using System.Linq;
using System.Threading.Tasks;

/// <summary>
/// Await all files in a list of paths to download
/// </summary>
public async Task DownloadFilesAsync(IWebClient client, IEnumerable<string> filePaths) 
{
    var downloadTasks = filePaths
        .Select(f => DownloadFileAsync(client, f))
        .ToArray();

    // if any fails, we return that an error occurred.  If you prefer to let "some" of
    // those downloads fail, just move the try/catch into the individual file download 
    // method below.
    try {
        // wait for all to complete
        await Task
            .WhenAll(downloadTasks)
            .ConfigureAwait(false);

        return Task.CompletedTask;
    } catch (Exception e) {
        // I just made this up as an example; find the right type 
        // of result using intellisense in your IDE
        return Task.ErroredTask(e);
    }
}

/// <summary>
/// Await a single file download
/// </summary>
public async Task DownloadFileTaskAsync(IWebClient client, string filePath) 
{
    // set up request in the client, etc
    var url = "http://example.com";

    return await client
        .DownloadFile(url, filePath)
        .ConfigureAwait(false);

}