Cast managed COM object to an interface without registering source assembly

Question

I'm rewriting a Word AddIn, which has the following basic structure:

MyAddIn exposes SetController method. This method is used to pass an instance of IMyController to the addin. It is then used to communicate between a C# app and the C# AddIn.

The MyAddIn.dll is registered for COM interop. It implements a RequestComAddInAutomationService (MS Office thing) that returns a COM object, which then allows the app to call the SetController method. It looks something like this:

using MyWordAddIn;
using System.Runtime.InteropServices;

namespace MyWordAddIn;

[ComVisible(true)]
[Guid(IAddInUtilities_GUID)]
public interface IAddInUtilities
{
    void SetController(object obj);
}

[ComVisible(true)]
[Guid(AddInUtilities_GUID)]
[ClassInterface(ClassInterfaceType.None)]
[ComSourceInterfaces(typeof(IAddInUtilities))]
public class AddInUtilities : StandardOleMarshalObject, IAddInUtilities
{
    public void SetController(object obj)
    {
        Globals.ThisAddIn.Controller = obj;
    }
}

Now in consumer app, when I get a COM instance of AddInUtilities, I can cast and use it like this:

var comInstance = ...;
var addInUtilities = (MyWordAddIn.IAddInUtilities)comInstance;
var ctrl = new Controller();
ctrl.Text = "test";
addInUtilities.SetController(ctrl);

With the Controller defined like this:

[ComVisible(true)]
[Guid(Controller_GUID)]
[ClassInterface(ClassInterfaceType.AutoDual)]
public class Controller
{
    public string Text { get; set; }
}

Now back in the AddIn, the .Controller is set to a COMObject instance, and I'm able to read the Text if I cast it to dynamic:

string txt = ((dynamic)this.Controller).Text;

I'd much rather cast to a class/interface instead, but I can't figure out how. I'm always getting an InvalidCastException, however I try to define an interface to cast to. I think if I register the app's dll it'll work, but I'd like to avoid it. In the worst case scenario, I can always use a wrapper around the dynamic, but I'd really like to avoid doing this.