How do I handle a base href when my project uses both angular and razor depending on the route you hit?

Question

I have a project which is using both mvc razor views and angular 8. Depending on the route that you hit, one or the other will be served. My routing is working for the most part and I am able to load my pages correctly. For now when users navigate to the home page, they will receive a razor view. In the console I get the error

No base href set. Please provide a value for the APP_BASE_HREF token or add a base element to the 
document.

The error itself doesn't seem to break anything, but is kind of annoying seeing it in all the logs (db, insights). It also makes debugging in IE a bit annoying too. Does anyone have any suggestions on how to fix it or maybe provide ideas on handling our situation a different way? If possible maybe the solution is somehow telling the application not to do anything with angular until the user hits the correct route?

We have a view called _Layout.cshtml (basically the entry page for the application). In this view we have some code:

@{ 
    var containsAngularRoute = Request.RawUrl.Contains("/app/");
    if (containsAngularRoute)
    {
        <base href="/" />
    }
  }

This was added because we noticed that without it, the angular site is just blank. And with it, we noticed some anchor tags breaking causing certain parts of the site not to function.

To enter angular we have a view with an index.cshtml file and it we just have:

<app-root></app-root>

The view has a controller that just returns the view:

[RoutePrefix("app")]
    public class AngularEntryController : Controller
    {
        public ActionResult Index()
        {
            return View();
        }
    }

The RouteConfig.cs file looks like this:

    routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

        // enables attributes routing
        routes.MapMvcAttributeRoutes();

        routes.MapRoute(
            name: "Default",
            url: "{controller}/{action}/{id}",
            defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional },
            constraints: new
            {
                serverRoute = new ServerRouteConstraint(url =>
                {
                    return !url.PathAndQuery.StartsWith("/app", StringComparison.InvariantCultureIgnoreCase);
                })
            });


        routes.MapRoute(
            name: "angular",
            url: "{*url}",
            defaults: new { controller = "AngularEntry", action = "Index" }
        );