How to find out which Task has finished

Question

I have an async code to upload all files in a directory to an API. It works fine but I would like to track which file has uploaded and would want to log, so that I can do some status check.

I have been following this guide and also this one . The first one creates explicit tasks for each file and the 2nd link, only talks about showing the number of tasks left. I, on the other hand, would like to find out which task has finished, print it to a log and carry on awaiting till there are no files to upload. I am using the Script component within SSIS to do this.

This is when 2 tasks are defined specifically for 2 files.

            Task<string> uploadCounterpartFileTask = UploadFiles(year, month, filename_counterparts, accesstoken, path);
            Task<string> uploadProductCategoryFileTask = UploadFiles(year, month, filename_productcategories, accesstoken, path);

            var allTasks = new List<System.Threading.Tasks.Task> { uploadCounterpartFileTask, uploadProductCategoryFileTask };


            while (allTasks.Any())
            {
                System.Threading.Tasks.Task finished = await System.Threading.Tasks.Task.WhenAny(allTasks);
                if (finished == uploadCounterpartFileTask)
                {
                    allTasks.Remove(uploadCounterpartFileTask);
                    var counterpartFile = await uploadCounterpartFileTask;
                }
                else if (finished == uploadProductCategoryFileTask)
                {
                    allTasks.Remove(uploadProductCategoryFileTask);
                    var productCategoriesFile = await uploadProductCategoryFileTask;
                }
            }

But I do not want to do that now as I would not know how many files would be in the directory

I would like to upload all files in a directory, something like this.:-

string[] fileSystemEntries = Directory.GetFileSystemEntries(path);

            var tasks = fileSystemEntries.OrderBy(s => s).Select(
            fileName => UploadFiles(year, month, Path.GetFileName(fileName), accesstoken, path));
            while (tasks.Any())
            {

                var finished = await System.Threading.Tasks.Task.WhenAny(tasks);



                    Dts.Events.FireInformation(0, "Script Task for uploading AVRO files", "File Uploaded" + finished.Result, string.Empty, 0, ref fireAgain);
                    tasks.Remove(finished);


            }
            Dts.Events.FireInformation(0, "Script Task for uploading AVRO files", "All Files Uploaded", string.Empty, 0, ref fireAgain);

I am using finished.Result because the documentation on MSDN states that Result gives the task that completed after whenAny. I know whenAny gives the first and therefore I am looping (similar to the examples on MSDN).

You see the Dts.Events.FireInformation as this is inside a script task within SSIS.

I would expect that it would say in the log something like.

File Uploaded: Counterparts.avro

File Uploaded: ProductCategorties.avro

File Uploaded: SomeOtherFile.avro

All Files uploaded.

But , though the file upload goes all right, none of the statements are printed.

How do I make them print? I just do not want to have a single notification for all files, I would really want to have a status of which file got uploaded.

EDIT: In response to one of the comments, here is the code for UploadFiles. Basically it uploads a file to an API and, per file involes 3 REST calls.

    private async Task<string> UploadFiles(string year, string month, string filename, string accesstoken, string path)
    {
        //Initial Put
        var method = new HttpMethod("PUT");
        string url = String.Format("https://someurl/part1/{0}/{1}/{2}?resource=file&recursive=True", year, month, filename);


        var request = new HttpRequestMessage(method, url)
        {
            Content = null
        };

        request.Headers.Add("Authorization", "Bearer " + accesstoken);


        var initput = await client.SendAsync(request);

        //Append Data
        url = String.Format("https://someurl/part1/{0}/{1}/{2}?action=append&position=0", year, month, filename);

        string SourceFile = path;
        var content = new MultipartFormDataContent();
        string filenamefullyqualified = path + filename;
        Stream fs = System.IO.File.Open(filenamefullyqualified, FileMode.Open, FileAccess.Read, FileShare.None);
        content.Add(CreateFileContent(fs, filename, "text/plain"));

        method = new HttpMethod("PATCH");

        request = new HttpRequestMessage(method, url)
        {
            Content = content
        };

        //long filelength = new System.IO.FileInfo(filenamefullyqualified).Length;

        request.Headers.Add("Authorization", "Bearer " + accesstoken);

        var response = await client.SendAsync(request);

        long? position = request.Content.Headers.ContentLength;

        //Flush Data

        url = String.Format("https://someurl/part1/{0}/{1}/{2}?action=flush&position={3}", year, month, filename, position.ToString());

        request = new HttpRequestMessage(method, url)
        {
            Content = null
        };

        request.Headers.Add("Authorization", "Bearer " + accesstoken);


        response = await client.SendAsync(request);


        return filename;

    }

EDIT 2 : In response to the comment. Please note that it is a sample code that I am trying, to demonstrate this problem.

    public async void Main()
    {
        bool fireAgain = true;
        HttpClientHandler httpClientHandler = new HttpClientHandler()
        {
            Proxy = new WebProxy("http://127.0.0.1:8888")
            {
                UseDefaultCredentials = true,
                BypassProxyOnLocal = true
            },
            UseDefaultCredentials = true
        };

        string year = "2019";
        string month = "02";
        string path = "D:\\SomeFolder\\AVRO\\";

        client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
        string resp = getBearerToken().GetAwaiter().GetResult();
        var ser = new JavaScriptSerializer();
        var result = ser.DeserializeObject(resp);
        Dictionary<string, object> BearerTokenDetails = new Dictionary<string, object>();
        BearerTokenDetails = (Dictionary<string, object>)(result);

        string accesstoken = BearerTokenDetails["access_token"].ToString();
        Dts.Events.FireInformation(0, "Script Task for uploading AVRO files", "Access Code: " + accesstoken, string.Empty, 0, ref fireAgain);


        string[] fileSystemEntries = Directory.GetFileSystemEntries(path);

        var tasks = fileSystemEntries.OrderBy(s => s).Select(
        fileName => UploadFiles(year, month, Path.GetFileName(fileName), accesstoken, path));

        await System.Threading.Tasks.Task.WhenAll(tasks);
        Dts.Events.FireInformation(0, "Script Task for uploading AVRO files", "All Files Uploaded", string.Empty, 0, ref fireAgain);


    }

You would see a getBearerToken method which I defined as async but later made it a Blocked call using GetResult as that needs to happen first. Also, point to note is- the first FireInfo that you see after the Bearer Token , only worked after I made it a blocking call (the GetResult does that as I read).

Answer 1

Assuming FireInformation works as intended, your code can be greatly simplified.

You can log the completion per task inside UploadFiles:

private async Task<string> UploadFiles(string year, string month, string filename, string accesstoken, string path)
{
    // stuff


    response = await client.SendAsync(request);

    // Log task finished
    Dts.Events.FireInformation(0, "Script Task for uploading AVRO files", "File Uploaded" + filename, string.Empty, 0, ref fireAgain);  

    return filename;
}

The end of the method is the moment the file has been uploaded and the task is finished. No need for a while loop and manually removing tasks from an array.

It think you even do not need to return the filename anymore if it is only for the logging

To log all completed message simply write a log file after Task.WhenAll:

        string[] fileSystemEntries = Directory.GetFileSystemEntries(path);

        var tasks = fileSystemEntries.OrderBy(s => s).Select(
        fileName => UploadFiles(year, month, Path.GetFileName(fileName), accesstoken, path));

        await Task.WhenAll(tasks);
        Dts.Events.FireInformation(0, "Script Task for uploading AVRO files", "All Files Uploaded", string.Empty, 0, ref fireAgain);

Now, since you are running in the console, you have two options, use public async Task Main() as the method signature (require c# 7.1, see this) or change await Task.WhenAll(tasks); to Task.WaitAll(tasks); otherwise the application won't wait for your code to finish and hence no output is shown.

Additionally an option is to add the line Console.ReadKey(); at the very end of your Main method and wait for the output to show.