CODEWITHSUNDEEP
3drealitykitstereo-3dvisionos
wcochran
wcochran

Reputation: 10896

Obtaining RealityKit view transforms for each eye used for stereo rendering

I am unable to find details on how RealityKit performs stereo rendering. I am used to systems that have two rendering pipelines -- one for each eye and I know what the view transforms are for each eye (typically computed as an x offset from the center monocular camera position based on the interocular distance). For example, in WebXR you know both the view and projection transform for each stereo eye:

let pose = xrFrame.getViewerPose(xrReferenceSpace);
...
for (let view of pose.views) {
    viewMatrix = view.transform.inverse.matrix;
    projectionMatrix = view.projectionMatrix;
    drawSceneWithCameraData(viewMatrix, projectionMatrix);
}

RealityKit, like modern Unity URP, evidently performs stereo rendering with a single pass, but there are still two different views for each eye. How can a retrieve stereo camera details for RealityKit on VisionOS?

Upvotes: 1

Views: 251

Answers (1)

Jakob
Jakob

Reputation: 982

you can get the view matrix for each eye using Compositor Services when you render inside a fully immersive space using Metal.

https://developer.apple.com/documentation/compositorservices/drawing_fully_immersive_content_using_metal

Apple seems very restrictive with giving out information about the user's pose that you can use in your app.

Compositor Service is the abstraction that is meant to be used if you do your own fully custom Metal rendering, and it gives you just the bare info that you need to know to get a perspective rendering going.

in RealityKit, since all you do is place objects into the world and don't touch the renderer, and therefore don't need the view transform, they don't give it to you, it seems.

In this other question you asked, you have found that you can access the eye index in RealityComposer shader graph: In Vision Pro's RealityKit how can I programmatically use a different texture map for each eye?

But it's strongly suggested that the SDK is designed in a way that abstracts the user from the app developer.

Upvotes: 1

Related Questions