VisionOS detect lateral head tilt

Question

Is it possible to detect a lateral head tilt within an Immersive Space?

When the AVP wearer tilts their head to the left or right, I want to detect it and trigger an animation.

As far as I can tell, it's not possible to replicate this movement in the Simulator. Is there a work-around to test it without owning a device?

I'm not sure if this is a viable input method as the Vision Pro has not been released in my country yet.

Answer 1

Here's how I ended up achieving a head roll trigger using RealityView and WorldTrackingProvider:

VisionProPose:

import ARKit
import QuartzCore
import Combine

class VisionProPose: ObservableObject {
    
    let arkitSession = ARKitSession()
    let worldTracking = WorldTrackingProvider()
    
    func runARKitSession() async {
        do {
            try await arkitSession.run([worldTracking])
        } catch {
            print("Failed to run ARKit session: \(error)")
            return
        }
    }

    func queryDeviceRoll() async -> Double? {
        
        // Wait until the worldTracking state is running
        while worldTracking.state != .running {
            try? await Task.sleep(nanoseconds: 100_000_000) // 100 milliseconds
        }
        
        let deviceAnchor = worldTracking.queryDeviceAnchor(atTimestamp: CACurrentMediaTime())
        
        if let deviceAnchor = deviceAnchor {
            let transform = deviceAnchor.originFromAnchorTransform
            
            // Extract the elements for the 2x2 submatrix
            let m00 = transform.columns.0.x
            let m10 = transform.columns.1.x
            
            // Compute the rotation around the Z-axis
            let zRotation = atan2(m10, m00)
            
            // Convert the rotation to degrees and then to Double
            let zRotationDegrees = Double(zRotation * 180.0 / .pi)
            
            return zRotationDegrees
        } else {
            print("No device anchor found")
            return nil
        }
    }
}

I then used it in a view like this:

ExampleView

import SwiftUI
import RealityKit

struct ExampleView: View {
    
    // Load vision pro pose
    @StateObject private var visionProPose = VisionProPose()
    
    var body: some View {
        RealityView { content in
            
            // Run each time scene is drawn (90Hz)
            _ =  content.subscribe(to: SceneEvents.Update.self) { event in
                
                Task {
                    
                    // Check the current head position
                    if let currentHeadRoll = await visionProPose.queryDeviceRoll() {
                        
                        // Head roll is more than 14 degrees in either direction
                        if abs(currentHeadRoll) > 14 {
                            
                            // Default to left
                            var tiltDirection = "left"
                            
                            // But could be right
                            if currentHeadRoll > 14 {
                                tiltDirection = "right"
                            }
                            
                            // Do something with head tilt
                            switch tiltDirection {
                            case "left": 
                                print("Head tilting left!")
                            case "right":
                                print("Head tilting right!")
                            default:
                                break
                            }
                        }
                    }
                }
            }
        }
    }
}

Together this code looks for a 14 degree head tilt, left or right and allows you to use it for whatever you want.

Answer 2

Head rolling detection

Tilting your head (camera) left or right called rolling in 3D graphics. At the moment, visionOS Simulator does not have a control for camera rotation around the Z axis. By the way, in the real world, this is not a rolling in its pure form, it's rather a rotation with an offset, since the pivot point around which the rotation occurs, is located not on the axis of the Vision Pro camera, but on the axis of the user’s neck.

To answer your question, I can say: yes, this is possible when using real Vision Pro. To do this, you'll need to run an ARKit session with WorldTrackingProvider. The output is a simd_float4x4 transform matrix, from which you'll have to take the four elements from the top left corner of the matrix (these 4 elements are responsible for rotation around the Z axis):

column.0.x,  column.0.y,  column.1.x,  column.1.y

Now you're able to detect which direction the user's head will be tilted in. For that, use simd_float4x4 extension to calculate Eulers from Quaternions, and then convert radians to degrees.

---

You can test it using the following code:

import SwiftUI
import RealityKit

extension simd_float4x4 {
    var eulerAngles: simd_float3 {
        simd_float3(
            x: asin(-self[2][1]),
            y: atan2(self[2][0], self[2][2]),
            z: atan2(self[0][1], self[1][1])
        )
    }
}
struct ContentView: View {
    let cube = ModelEntity(mesh: .generateBox(size: 0.5))

    var body: some View {
        RealityView { rvc in
            cube.orientation = .init(angle: -.pi/4, axis: [0,0,1])
            rvc.add(cube)
            
            let eulerZ = cube.transform.matrix.eulerAngles.z
            let degrees = rad2deg(eulerZ)
            print(degrees, "degrees")     // prints: -45.0 degrees
        }
    }
    func rad2deg(_ value: Float) -> Float {
        value * 180.0 / Float.pi
    }
}