Akka.Net not executing await continuation in PostStop

Question

My actor interacts with a non-Akka thing that has an async disposal. This disposal can take 5-10 seconds. I do this in PostStop() like so:

protected override void PostStop()
{
    async Task DisposeThing()
    {
        Debug.WriteLine("Before Delay");
        await Task.Delay(10000); // This would be the actual call to dispose the thing
        Debug.WriteLine("After Delay");
    };

    ActorTaskScheduler.RunTask(async () =>
    {
        try
        {
            Debug.WriteLine("Before DisposeThing");
            await DisposeThing();
            Debug.WriteLine("After DisposeThing");
        }
        catch (Exception ex)
        {
            Debug.WriteLine($"An exception occured: {ex}");
        }
        finally
        {
            Debug.WriteLine("actor done disposing.");
        }
    });
    base.PostStop();
}

Full gist here.

The parent does _childActor.Tell(PoisonPill.Instance). I also tried _childActor.GracefulStop with a large enough timeout.

In both cases, this prints:

Before DisposeThing
Before Delay

And that's it, the rest is never executed. Not even the finally executes (which I guess breaks C#? using doesn't work anymore, for instance).

Silently dropping await continuations (including finallys) could lead to some really tricky-to-understand bugs, so this leaves me with two questions:

1. when does Akka decide to simply drop an ongoing async function, is there a consistent model to be understood?
2. how should I write this in a way that is guaranteed to execute and not terminate the actor before disposal is done?

Update: After sleeping on this I think I understand what's going on. Keep in mind that is mostly conjecture from someone that's been looking at Akka.Net for the past 2 days (e.g. this thread), and I post this because no one has answered yet.

The way Akka.Net implements async continuations is by having the actor executing the async function send ActorTaskSchedulerMessages to itself. This message points to the remaining work to be done after an await returns, and when the actor gets to process that message, it'll execute the continuation, up until the next await (or the end of the function if there no other await).

When you tell an actor to stop with a PoisonPill for instance, once that message is processed, no further messages are processed for that actor. This is fine when those messages are user-defined. However, this ends up also silently dropping any async continuations since they're also implemented as actor messages.

Indeed when running a program using the above actor, we can see this in the console:

[INFO][2022-01-11 2:59:43 PM][Thread 0004][akka://ActorSystem/user/$a/$a] Message [ActorTaskSchedulerMessage] from [akka://ActorSystem/user/$a/$a#132636847] to [akka://ActorSystem/user/$a/$a#132636847] was not delivered. 2 dead letters encountered. If this is not an expected behavior then [akka://ActorSystem/user/$a/$a#132636847] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

If this understanding is correct, this makes async extremely unreliable inside functions passed to ReceiveAsync, ActorTaskScheduler.RunTask etc. as you cannot ever assume anything after an await will get to execute, including exception handlers, cleanup code inside finallys, using statement disposal, etc. Actors can be killed stopped at any time.

I suppose then that since language primitives lose their meaning, what you need to do is wrap your Task-returning functions inside their own little actors and rely on Akka semantics rather than language semantics.

Answer 1

You captured what the issue was.

This is one solution I came up with:

using Akka.Actor;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using static Akka.NET_StackOverflow_Questions_tryout.Questions._70655287.ChildActor;

namespace Akka.NET_StackOverflow_Questions_tryout.Questions._70655287
{
    public class ParentActor:ReceiveActor
    {
        private readonly IActorRef _child;
        public ParentActor()
        {
            _child = Context.ActorOf(ChildActor.Prop());
            Context.Watch(_child);
            
            Receive<ShutDown>(s =>
            {
                _child.Forward(s);
            });
            Receive<Terminated>(t => {
                var tt = t;
            });  
        }
        public static Props Prop()
        {
            return Props.Create(() => new ParentActor());
        }
    }


    public class ChildActor : ReceiveActor
    {
        public ChildActor()
        {
            ReceiveAsync<ShutDown>(async _ =>
            {
                async Task DisposeThing()
                {
                    Debug.WriteLine("Before Delay");
                    await Task.Delay(10000); // This would be the actual call to dispose the thing
                    Debug.WriteLine("After Delay");
                };
                await DisposeThing()
                .ContinueWith(async task => 
                {
                    if (task.IsFaulted || task.IsCanceled)
                        return; //you could notify the parent of this issue
                    await Self.GracefulStop(TimeSpan.FromSeconds(10));
                });
            });
        }
        protected override void PostStop()
        {            
            base.PostStop();
        }
        public static Props Prop()
        {
            return Props.Create(()=> new ChildActor());
        }

        public sealed class ShutDown
        {
            public static ShutDown Instance => new ShutDown();  
        }
    }
}

So instead of stopping the _childActor from the parentActor side you could send a shutdown message to the child to shutdown following the defined steps: first dispose the non-akka thing (to ensure it is truly not alive in-memory) afterwards, second, self-destruct the child which will notify the parent!