How to check if face in full visible and in center of screen with ARFaceAnchor and RealityKit?

Question

I'm using ARKit and RealityKit with an ARFaceAnchor object. I would like to ensure that the person's face is close or near the bounds of the ellipse in the center of the device screen.

I'm not seeing how to get the dimensions of the face, like a minX/maxX and minY/maxY, and how to translate those values to the coordinates of the screen, specifically where the ellipse shape is located.

Can anyone point to a solution?

Answer 1

I implemented a similar approach in one of my apps to track the users face. I was using it for ARSCNView and SceneKit, but it should also work for ARView and RealityKit (probably you need to do some small adjustments)

Implement this and adapt it to your logic.

The function prints out the face position for each frame. You can then implement some boundary checks or whatever to keep the face centered, or alert the user to move closer to the center.

extension ViewController {
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        guard let faceAnchor = anchors.first as? ARFaceAnchor else { return }
        checkFacePosition(faceAnchor: faceAnchor)
    }

    private func checkFacePosition(faceAnchor: ARFaceAnchor) {
        guard let frame = arSession.currentFrame else { return }

        // Combine the camera's transform with the face transform
        let faceTransform = faceAnchor.transform
        let cameraTransform = frame.camera.transform
        let combinedTransform = cameraTransform.inverse * faceTransform

        // Extract the face's translation (position relative to the camera)
        let translation = combinedTransform.columns.3

        // Normalize offsets using a reference distance
        let referenceDistance: Float = 0.5
        let xOffset = translation.y / referenceDistance // Flip x/y to match screen orientation
        let yOffset = translation.x / referenceDistance

        // Define thresholds for an ellipse (values can be adjusted)
        let ellipseWidth: Float = 0.2 // Adjust for the horizontal size of the ellipse
        let ellipseHeight: Float = 0.3 // Adjust for the vertical size of the ellipse

        // Check if the face is within the ellipse bounds
        let isInsideEllipse = pow(xOffset, 2) / pow(ellipseWidth / 2, 2) + pow(yOffset, 2) / pow(ellipseHeight / 2, 2) <= 1.0

        // Log the result
        if isInsideEllipse {
            print("Face is inside the ellipse: xOffset: \(xOffset), yOffset: \(yOffset)")
        } else {
            print("Face is outside the ellipse: xOffset: \(xOffset), yOffset: \(yOffset)")
        }
    }
}

This are some print-out's from the console logger:

Face is outside the ellipse: xOffset: 0.026052857, yOffset: -0.2010466
Face is outside the ellipse: xOffset: 0.018577524, yOffset: -0.18251653
Face is outside the ellipse: xOffset: 0.009894665, yOffset: -0.16267225
Face is inside the ellipse: xOffset: 0.0024384973, yOffset: -0.13900988
Face is inside the ellipse: xOffset: 0.002202771, yOffset: -0.11147189
Face is inside the ellipse: xOffset: 0.008226621, yOffset: -0.08980697

You can quickly check it out, by setting up a default ARKit project with SceneKit Engine. Then replace your ViewController with that code

Scenekit Solution

class ViewController: UIViewController, ARSCNViewDelegate, ARSessionDelegate {

    @IBOutlet var sceneView: ARSCNView!
    private var arSession: ARSession {
        return sceneView.session
    }

    override func viewDidLoad() {
        super.viewDidLoad()

        // Set the view's delegate
        sceneView.delegate = self
        sceneView.session.delegate = self

        // Show statistics such as fps and timing information
        sceneView.showsStatistics = true

        // Create a new scene
        let scene = SCNScene()

        // Set the scene to the view
        sceneView.scene = scene
    }

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        // Create a session configuration
        guard ARFaceTrackingConfiguration.isSupported else {
            print("ARFaceTracking is not supported on this device.")
            return
        }

        let configuration = ARFaceTrackingConfiguration()
        configuration.isLightEstimationEnabled = true // Enable light estimation if needed

        // Run the view's session with face tracking
        sceneView.session.run(configuration)
    }

    override func viewWillDisappear(_ animated: Bool) {
        super.viewWillDisappear(animated)

        // Pause the view's session
        sceneView.session.pause()
    }

    // MARK: - ARSCNViewDelegate (Optional Node Configuration)

    func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
        guard anchor is ARFaceAnchor else { return nil }

        // Return a new node for face anchor (optional for visualization)
        let node = SCNNode()
        node.geometry = SCNSphere(radius: 0.02)
        node.geometry?.firstMaterial?.diffuse.contents = UIColor.red
        return node
    }

    func session(_ session: ARSession, didFailWithError error: Error) {
        // Present an error message to the user
        print("ARSession failed with error: \(error.localizedDescription)")
    }

    func sessionWasInterrupted(_ session: ARSession) {
        // Inform the user that the session has been interrupted
        print("ARSession was interrupted.")
    }

    func sessionInterruptionEnded(_ session: ARSession) {
        // Reset tracking if consistent tracking is required
        print("ARSession interruption ended.")
    }
}

I hope this will help you to achieve what you are looking for, or at least point you in some direction.

PS: You will need a device with LIDAR sensor support