Identifying when long running async tasks have completed

Question

[Updated 18-Apr-2018 with LinqPad example - see end]

My application receives a list of jobs:

var jobs = await myDB.GetWorkItems();

(NB: we use .ConfigureAwait(false) everwhere, I'm just not showing it in these pseudo code snippets.)

For each job, we create a long running Task. However, we don't want to wait for this long running Task to complete.

jobs.ForEach(job =>
{
    var module = Factory.GetModule(job.Type);
    var task = Task.Run(() => module.ExecuteAsync(job.Data));
    this.NonAwaitedTasks.Add(task, module);
};

The task and its related module instance are both added to a ConcurrentDictionary so that they don't go out of scope.

Elsewhere, I have another method that is called occasionally which contains the following:

foreach (var entry in this.NonAwaitedTasks.Where(e => e.Key.IsCompleted))
{
    var module = entry.Value as IDisposable;
    module?.Dispose();
    this.NonAwaitedTasks.Remove(entry.Key);
}

(NB the NonAwaitedTasks is additionally locked using a SemaphoreSlim...)

So, the idea is that this method will find all those Tasks which have completed and then dispose of their related module, and remove them from this Dictionary.

However....

Whilst debugging in Visual Studio 2017, I pull a single job from the DB and whilst I'm taking my time debugging within the single Module that has been instantiated, the Dispose is called on that module. Looking in the Callstack, I can see the Dispose has been called in the method above, and that is because the task has IsCompleted == true. But evidently, it can't be completed because I'm still debugging it.

• Is the .IsCompleted property the wrong property to check?
• Is this just an artifact of debugging in Visual Studio?
• Am I going about this the wrong way?

Additional Information

In the comments below, I was asked to provide some additional information regarding the flow because what I described didn't seem possible (and indeed, my hope was that it couldn't be). Below is a cut-down version of my code (I've removed checks for the cancellation token and defensive coding, but nothing that affects the flow).

Application Entry Point

This is a Windows Service. In the OnStart() is the following line:

this.RunApplicationTask = 
Task.Run(() => myApp.DoWorkAsync().ConfigureAwait(false), myService.CancelSource.Token);

"RunApplicationTask" is just a property to keep the executing task in scope during the lifetime of the Service.

DoWorkAsync()

public async Task DoWorkAsync()
{
    do
    {
        await this.ExecuteSingleIterationAsync().ConfigureAwait(false);
        await Task.Delay(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
    }
    while (myApp.ServiceCancellationToken.IsCancellationRequested == false);

    await Task.WhenAll(this.NonAwaitedTasks.Keys).ConfigureAwait(false);
    await this.ClearCompletedTasksAsync().ConfigureAwait(false);
    this.WorkItemsTaskCompletionSource.SetResult(true);

    return;
}

So whilst I'm debugging, this is iterating the DO-LOOP, it does not get to the Task.WhenAll(....).

Note too that after the Cancellation request is called and all Tasks have completed, I call ClearCompletedTasksAsync(). More on that later....

ExecuteSingleIterationAsync

private async Task ExecuteSingleIterationAsync()
{
    var getJobsResponse = await DB.GetJobsAsync().ConfigureAwait(false);
    await this.ProcessWorkLoadAsync(getJobsResponse.Jobs).ConfigureAwait(false);
    await this.ClearCompletedTasksAsync().ConfigureAwait(false);
}

ProcessWorkLoadAsync

private async Task ProcessWorkLoadAsync(IList<Job> jobs)
{
    if (jobs.NoItems())
    {
        return ;
    }

    jobs.ForEach(job =>
    {
        // The processor instance is disposed of when removed from the NonAwaitedTasks collection.
        IJobProcessor processor = ProcessorFactory.GetProcessor(workItem, myApp.ServiceCancellationToken);
        try
        {
            var task = Task.Run(() => processor.ExecuteAsync(job).ConfigureAwait(false), myApp.ServiceCancellationToken);
            this.NonAwaitedTasks.Add(task, processor);
        }
        catch (Exception e)
        {
            ...
        }
    });

    return;
}

Each processor implements the following interface method: Task ExecuteAsync(Job job);

It's whilst I'm in the ExecuteAsync that .Dispose() gets called on the processor instance I'm using.

ProcessorFactory.GetProcessor()

public static IJobProcessor GetProcessor(Job job, CancellationToken token)
{
    .....
    switch (someParamCalculatedAbove)
    {
        case X:
            {
                return new XProcessor(...);
            }

        case Y:
            {
                return new YProcessor(...);
            }

        default:
            {
                return null;
            }
    }
}

So here we're getting a new instance.

ClearCompletedTasksAsync()

private async Task ClearCompletedTasksAsync()
{
    await myStatic.NonAwaitedTasksPadlock.WaitAsync().ConfigureAwait(false);
    try
    {
        foreach (var taskEntry in this.NonAwaitedTasks.Where(entry => entry.Key.IsCompleted).ToArray())
        {
            var processorInstance = taskEntry.Value as IDisposable;
            processorInstance?.Dispose();
            this.NonAwaitedTasks.Remove(taskEntry.Key);
        }
    }
    finally
    {
        myStatic.NonAwaitedTasksPadlock.Release();
    }
}

This is called every iteration of the Do-Loop. It's purpose is to ensure that the list of non-awaited tasks is kept small.

And that's it... Dispose only seems to get called when debugging.

LinqPad example

async Task Main()
{
    SetProcessorRunning();

    await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false);

    do
    {
        foreach (var entry in NonAwaitedTasks.Where(e => e.Key.IsCompleted).ToArray())
        {
            "Task is completed, so will dispose of the Task's processor...".Dump();
            var p = entry.Value as IDisposable;
            p?.Dispose();
            NonAwaitedTasks.Remove(entry.Key);
        }
    }
    while (NonAwaitedTasks.Count > 0);
}

// Define other methods and classes here

public void SetProcessorRunning()
{
    var p = new Processor();

    var task = Task.Run(() => p.DoWorkAsync().ConfigureAwait(false));
    NonAwaitedTasks.Add(task, p);
}

public interface IProcessor
{
    Task DoWorkAsync();
}

public static Dictionary<Task, IProcessor> NonAwaitedTasks = new Dictionary<Task, IProcessor>();

public class Processor : IProcessor, IDisposable
{
    bool isDisposed = false;
    public void Dispose()
    {
        this.isDisposed = true;
        "I have been disposed of".Dump();
    }

    public async Task DoWorkAsync()
    {
        await Task.Delay(TimeSpan.FromSeconds(5)).ConfigureAwait(false);

        if (this.isDisposed)
        {
            $"I have been disposed of (isDispose = {this.isDisposed}) but I've not finished work yet...".Dump();
        }

        await Task.Delay(TimeSpan.FromSeconds(5)).ConfigureAwait(false);
    }
}

Output:

Task is completed, so will dispose of the Task's processor...

I have been disposed of

I have been disposed of (isDispose = True) but I've not finished work yet...

Answer 1

Your problem is in this line:

var task = Task.Run(() => p.DoWorkAsync().ConfigureAwait(false));

Hover over the var and take a look at what type that is.

Task.Run understands async delegates by having special "task unwrapping" rules for Func<Task<Task>> and friends. But it won't have any special unwrapping for Func<ConfiguredTaskAwaitable>.

You can think of it this way; with the code above:

1. p.DoWorkAsync() returns a Task.
2. Task.ConfigureAwait(false) returns a ConfiguredTaskAwaitable.
3. So the Task.Run is being asked to run this function that creates a ConfiguredTaskAwaitable on a thread pool thread.
4. Thus, the return type of Task.Run is Task<ConfiguredTaskAwaitable> - a task that completes as soon as the ConfiguredTaskAwaitable is created. When it is created - not when it completes.

In this case, the ConfigureAwait(false) isn't doing anything anyway, because there's no await to configure. So you can remove it:

var task = Task.Run(() => p.DoWorkAsync());

Also, as Servy mentioned, if you don't need to run DoWorkAsync on a thread pool thread, you can also skip the Task.Run:

var task = p.DoWorkAsync();