How to enable eye tracking for Hololens2 using MRTK3

Question

I am a beginner and have just started working with Unity 2022, Hololens2 and MRTK3. For my project, I am trying to enable gaze interaction such that when a person gazes at an object(eg. a sphere hologram) for more than a threshold amount of time(eg. 3 seconds), a text(textMeshPro) that was previously invisible becomes visible due to the gaze. I would really appreciate it if someone could guide me through all the necessary steps. I understand that I am supposed to use Gaze Controller but I cannot figure out how.

I have been looking for tutorials/ step-by-step examples online, but with no luck. Most of the content I have found so far is related to MRTK2, which doesn't help much given the changes.

Thanks in advance!

Answer 1

Resources on MRTK3 are indeed somewhat scarse. A good starting point is the MRTK3 dev template. In the following let's try to recreate your example use case. I am assuming that you got a basic scene using the MRTK XR Rig from MRTK3 running.

First, you need a reference to the FuzzyGazeInteractor e.g. as gazeInteractor. You will probably find it under MRTK XR Rig -> MRTK Gaze Controller -> GazeInteractor. The following script then performs raycasting into the scene and checks for collision with e.g. the sphere, let's call it targetObj. The gaze interactor provides us with some handy information such as rayOriginTransform which we can use to determine the gaze direction and perform raycasting. In your use case you want to trigger sth after the user gazed upon something for gazeTriggerDuration seconds. We sum the elapsed time between frames if collision with targetObject is detected, otherwise we reset. (Disclaimer: This code is not tested)

// References on script to gaze interactor and target object
public FuzzyGazeInteractor gazeInteractor;
public GameObject targetObj;

// Max. distance used for raycasting
public float rayMaxDistance = 5f;

// Timer for gaze duration
public float gazeTriggerDuration = 3f;
// Elapsed time between collision
private float elapsedTime = 0f;

// Check for gaze collision continuously
void Update()
{
    // Create ray along gaze direction
    var gazeRay = new Ray(gazeInteractor.rayOriginTransform.position, gazeInteractor.rayOriginTransform.forward * rayMaxDistance);
    if (Physics.Raycast(gazeRay, out var hit))
    {
        // Check if we hit target object or sth else (could also use tags)
        if (hit.collider.gameObject == targetObj)
        {
            // Increase elapsed time
            elapsedTime += Time.deltaTime;

            // Check if gaze trigger duration surpassed
            if (elapsedTime >= gazeTriggerDuration)
            {
                // Do sth e.g. enable text obj
                ...
            }
        }
        else
        {
            // No collision with target object
            elapsedTime = 0f;
        }
    }
    else
    {
        // No collision at all
        elapsedTime = 0f;
    }
}

As a side note: If you are interested in gaze interaction with UI elements such as buttons, some components come with built-in MRTK Events, including events such as Is Gaze Hovered which you can use to trigger callbacks etc. See the following image for some reference: