How to run an inner "ServerT" with different context to the outer "ServerT"

Question

We are writing custom servant combinators to handle various authorisation layers in our application. E.g. if you had a task management system, you might have an outermost scope where no authentication has been done, then the first layer would handle user authentication (which might give you access to some resources), a further layer might check your authorisation for a "project", which would create a new context, and so on.

The problem I have run into is that serveWithContext transforms the ServerT into a WAI.Application, which does not compose into layers in this way as we have broken out of servant-world entirely.

I think what I want to do looks something like this:

serveWithContext' ... (myOuterContext) $
  \userInfo -> serveWithContext' ... (mkNewContext userInfo) $
   ...

I can't work out the shape of the functions needed for this, however.

Motivation

The reason I need this to be constructed in layers is that we are using the ReaderT over IO pattern, and we construct different operational contexts at each layer (as parts of the application then can use mtl-style constraints to state their requirements). I cannot construct the inner operational contexts until I have passed through the various outer routing layers (like authentication, which adds user data to the context, or "project authorisation" which adds project data to the context, etc.).

This isn't to pass data to the handlers themselves, that's easy enough, but actually to pass server contexts to the inner combinators which they need to be able to run the checks in the route definition.

Current attempt

My current attempt at this looks a bit like this:

class Layer a where
  type family InnerContext a

  runLayer ::
    forall api ctx ctx' m env.
    ( HasServer (a :> api) ctx,
      HasServer api ctx',
      HasContextEntry ctx' (InnerContext a)
    ) =>
    Proxy a -> -- the outer api proxy
    Proxy api -> -- the inner api proxy
    (m a -> Handler a) -> -- the hoist
    ServerT api m -> -- the inner server to run
    Router env

I thought I had worked out the direction needed, but am now not even sure if it is possible.