How to prevent an application from terminating before the completion of all fire-and-forget tasks?

Question

I have an application that regularly launches fire-and-forget tasks, mainly for logging purposes, and my problem is that when the application is closed, any currently running fire-and-forget tasks are aborted. I want to prevent this from happening, so I am searching for a mechanism that will allow me to await the completion of all running fire-and-forget operations before closing my app. I don't want to handle their possible exceptions, I don't care about these. I just want to give them the chance to complete (probably with a timeout, but this is not part of the question).

You could argue that this requirement makes my tasks not truly fire-and-forget, and there is some truth in that, so I would like to clarify this point:

1. The tasks are fire-and-forget locally, because the method that launches them is not interested about their outcome.
2. The tasks are not fire-and-forget globally, because the application as a whole cares about them.

Here is a minimal demonstration of the problem:

static class Program
{
    static async Task Main(string[] args)
    {
        _ = Log("Starting"); // fire and forget
        await Task.Delay(1000); // Simulate the main asynchronous workload
        CleanUp();
        _ = Log("Finished"); // fire and forget

        // Here any pending fire and forget operations should be awaited somehow
    }

    private static void CleanUp()
    {
        _ = Log("CleanUp started"); // fire and forget
        Thread.Sleep(200); // Simulate some synchronous operation
        _ = Log("CleanUp completed"); // fire and forget
    }

    private static async Task Log(string message)
    {
        await Task.Delay(100); // Simulate an async I/O operation required for logging
        Console.WriteLine($"{DateTime.Now:HH:mm:ss.fff} {message}");
    }
}

Output:

11:14:11.441 Starting
11:14:12.484 CleanUp started
Press any key to continue . . .

The "CleanUp completed" and "Finished" entries are not logged, because the application terminates prematurely, and the pending tasks are aborted. Is there any way that I can await them to complete before closing?

Btw this question is inspired by a recent question by @SHAFEESPS, that was sadly closed as unclear.

Clarification: The minimal example presented above contains a single type of fire-and-forget operation, the Task Log method. The fire-and-forget operations launched by the real world application are multiple and heterogeneous. Some even return generic tasks like Task<string> or Task<int>.

It is also possible that a fire-and-forget task may fire secondary fire-and-forget tasks, and these should by allowed to start and be awaited too.

Answer 1

One reasonable thing is to have in-memory queue inside your logger (this applies to other similar functionality matching your criterias), which is processed separately. Then your log method becomes just something like:

private static readonly BlockingCollection<string> _queue = new BlockingCollection<string>(new ConcurrentQueue<string>());
public static void Log(string message) {
   _queue.Add(message);
}

It's very fast and non-blocking for the caller, and is asynchronous in a sense it's completed some time in the future (or fail). Caller doesn't know or care about the result, so it's a fire-and-forget task.

However, this queue is processed (by inserting log messages into final destination, like file or database) separately, globally, maybe in a separate thread, or via await (and thread pool threads), doesn't matter.

Then before application exit you just need to notify queue processor that no more items are expected, and wait for it to complete. For example:

_queue.CompleteAdding(); // no more items
_processorThread.Join(); // if you used separate thread, otherwise some other synchronization construct.

EDIT: if you want for queue processing to be async - you can use this AsyncCollection (available as nuget package). Then your code becomes:

class Program {
    private static Logger _logger;
    static async Task Main(string[] args) {
        _logger = new Logger();
        _logger.Log("Starting"); // fire and forget
        await Task.Delay(1000); // Simulate the main asynchronous workload
        CleanUp();
        _logger.Log("Finished"); // fire and forget
        await _logger.Stop();
        // Here any pending fire and forget operations should be awaited somehow
    }

    private static void CleanUp() {
        _logger.Log("CleanUp started"); // fire and forget
        Thread.Sleep(200); // Simulate some synchronous operation
        _logger.Log("CleanUp completed"); // fire and forget
    }
}

class Logger {
    private readonly AsyncCollection<string> _queue = new AsyncCollection<string>(new ConcurrentQueue<string>());
    private readonly Task _processorTask;
    public Logger() {
        _processorTask = Process();
    }

    public void Log(string message) {
        // synchronous adding, you can also make it async via 
        // _queue.AddAsync(message); but I see no reason to
        _queue.Add(message);
    }

    public async Task Stop() {
        _queue.CompleteAdding();
        await _processorTask;
    }

    private async Task Process() {
        while (true) {
            string message;
            try {
                message = await _queue.TakeAsync();
            }
            catch (InvalidOperationException) {
                // throws this exception when collection is empty and CompleteAdding was called
                return;
            }

            await Task.Delay(100);
            Console.WriteLine($"{DateTime.Now:HH:mm:ss.fff} {message}");
        }
    }
}

Or you can use separate dedicated thread for synchronous processing of items, as usually done.

EDIT 2: here is variation of reference counting which doesn't make any assumptions about nature of "fire and forget" tasks:

static class FireAndForgetTasks {

    // start with 1, in non-signaled state
    private static readonly CountdownEvent _signal = new CountdownEvent(1);

    public static void AsFireAndForget(this Task task) {
        // add 1 for each task
        _signal.AddCount();
        task.ContinueWith(x => {
            if (x.Exception != null) {
                // do something, task has failed, maybe log 
            }
            // decrement 1 for each task, it cannot reach 0 and become signaled, because initial count was 1
            _signal.Signal();
        });
    }

    public static void Wait(TimeSpan? timeout = null) {
        // signal once. Now event can reach zero and become signaled, when all pending tasks will finish
        _signal.Signal();
        // wait on signal
        if (timeout != null)
            _signal.Wait(timeout.Value);
        else
            _signal.Wait();
        // dispose the signal
        _signal.Dispose();
    }
}

Your sample becomes:

static class Program {
    static async Task Main(string[] args) {
        Log("Starting").AsFireAndForget(); // fire and forget
        await Task.Delay(1000); // Simulate the main asynchronous workload
        CleanUp();
        Log("Finished").AsFireAndForget(); // fire and forget
        FireAndForgetTasks.Wait();
        // Here any pending fire and forget operations should be awaited somehow
    }

    private static void CleanUp() {
        Log("CleanUp started").AsFireAndForget(); // fire and forget
        Thread.Sleep(200); // Simulate some synchronous operation
        Log("CleanUp completed").AsFireAndForget(); // fire and forget
    }

    private static async Task Log(string message) {
        await Task.Delay(100); // Simulate an async I/O operation required for logging
        Console.WriteLine($"{DateTime.Now:HH:mm:ss.fff} {message}");
    }
}

Answer 2

Perhaps something like a counter to wait on exit? This would still be pretty much fire and forget.

I only moved LogAsync to it's own method as to not need the discard every time Log is called. I suppose it also takes care of the tiny race condition that would occur if Log was called just as the program exited.

public class Program
{
    static async Task Main(string[] args)
    {
        Log("Starting"); // fire and forget
        await Task.Delay(1000); // Simulate the main asynchronous workload
        CleanUp();
        Log("Finished"); // fire and forget

        // Here any pending fire and forget operations should be awaited somehow
        var spin = new SpinWait();

        while (_backgroundTasks > 0)
        {
            spin.SpinOnce();
        }
    }

    private static void CleanUp()
    {
        Log("CleanUp started"); // fire and forget
        Thread.Sleep(200); // Simulate some synchronous operation
        Log("CleanUp completed"); // fire and forget
    }

    private static int _backgroundTasks;

    private static void Log(string message)
    {
        Interlocked.Increment(ref _backgroundTasks);
        _ = LogAsync(message);
    }

    private static async Task LogAsync(string message)
    {
        await Task.Delay(100); // Simulate an async I/O operation required for logging
        Console.WriteLine($"{DateTime.Now:HH:mm:ss.fff} {message}");
        Interlocked.Decrement(ref _backgroundTasks);
    }
}