Hololens 2 & Unity - How to use the CameraToWorldMatrix to correct the position of a hologram?

Question

I'm using the Hololens 2 with Unity 2019.4.6.

With the PhotoCaptureFrame class, I'm taking a picture, analyzing it to find a specific marker and then I create an hologram where the marker is. My code is working well with a webcam in the Unity Editor. But on the hololens, there is an offset between the hologram and the marker.

I try to correct it with cameraToWorld matrix (that I get with the photocaptureframe class). If I understand well, it represents the offset between the hololens camera and the RGB camera (the one taking a picture).

The problem is : I don't know if I'm using well this matrix, my hologram position is always weird... I tried to use this tutorial : https://forum.unity.com/threads/holographic-photo-blending-with-photocapture.416023/ And the result is wrong too ! The instantiated quad is rotated and behind the original starting point (when I start the app). It doesn't follow the camera.

Do I need to convert the matrix ? Can someone explains to me what I'm doing wrong please ? Thanks in advance !

Simple example to get matrices : It prints out true for the CameraToWorld matrix and The Projection matrix but the matrices are not updated when you move and take a picture from another location.

using System;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Windows.WebCam;

public class HoloLensSnapshotTest : MonoBehaviour
{
    PhotoCapture m_PhotoCaptureObj;
    CameraParameters m_CameraParameters;
    bool m_CapturingPhoto = false;

    void Start()
    {
        Initialize();
    }


    [ContextMenu("TakePicture")]
    public void TakePicture() // called by button for now
    {
        if (m_CapturingPhoto)
        {
            return;
        }

        m_CapturingPhoto = true;
        Debug.Log("Taking picture...");
        m_PhotoCaptureObj.TakePhotoAsync(OnPhotoCaptured);
    }

    void Initialize()
    {
        Debug.Log("Initializing...");
        List<Resolution> resolutions = new List<Resolution>(PhotoCapture.SupportedResolutions);
        Resolution selectedResolution = resolutions[0];

        m_CameraParameters = new CameraParameters(WebCamMode.PhotoMode);
        m_CameraParameters.cameraResolutionWidth = selectedResolution.width;
        m_CameraParameters.cameraResolutionHeight = selectedResolution.height;
        m_CameraParameters.hologramOpacity = 0.0f;
        m_CameraParameters.pixelFormat = CapturePixelFormat.BGRA32;

        PhotoCapture.CreateAsync(false, OnCreatedPhotoCaptureObject);
    }

    void OnCreatedPhotoCaptureObject(PhotoCapture captureObject)
    {
        m_PhotoCaptureObj = captureObject;
        m_PhotoCaptureObj.StartPhotoModeAsync(m_CameraParameters, OnStartPhotoMode);
    }

    void OnStartPhotoMode(PhotoCapture.PhotoCaptureResult result)
    {
        m_CapturingPhoto = false;

        Debug.Log("Ready");
    }


    void OnPhotoCaptured(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
    {
        Matrix4x4 cameraToWorldMatrix;
        bool cameraToWorldMatrixResult = photoCaptureFrame.TryGetCameraToWorldMatrix(out cameraToWorldMatrix);

        Matrix4x4 projectionMatrix;
        bool projectionMatrixResult = photoCaptureFrame.TryGetProjectionMatrix(out projectionMatrix);

        Debug.Log("CamToWorld : " + Environment.NewLine + cameraToWorldMatrixResult + Environment.NewLine + $"{cameraToWorldMatrix.ToString()}");
        Debug.Log("Projection : " + Environment.NewLine + projectionMatrixResult + Environment.NewLine + $"{projectionMatrix.ToString()}");
        Debug.Log("Took picture!");

        m_CapturingPhoto = false;
    }
}

My matrix are : Camera To World :

0.00650   -0.99959  -0.02805    0.00211
-0.99965  -0.00577  -0.02588   -0.04999
0.02571    0.02821  -0.99927   -0.01216
0.00000    0.00000   0.00000    1.00000

Projection :

-0.06754   1.52561   0.00000  0.00000
-2.71531  -0.12021   0.00000  0.00000
0.05028    0.01975  -1.00401 -0.20040
0.00000    0.00000   0.00000  1.00000

Answer 1

Good news, this has been fixed via Unity 2019.4.13f1 release on October 21st, 2020:

https://unity3d.com/unity/qa/lts-releases?version=2019.4

"XR: Fixed rotated projection when using PhotoCaptureFrame on HoloLens 2"

Thanks, Nathan

Answer 2

Actually, in an early version of the MR documentation, it has demonstrated how to find or draw at a specific 3d location on a camera image with shader code: mixed-reality-docs/locatable-camera.md.

But now this part of the document has been deleted. Fortunately, I found a solution on Github that implements the same function in Unity C# and only modified a few lines of code:cameraToWorld.cs

The following is a quote from the solution, you can programmatically store this data in your project. If you have any other question, please let me know:

void OnPhotoCaptured(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
    Matrix4x4 cameraToWorldMatrix;
    photoCaptureFrame.TryGetCameraToWorldMatrix(out cameraToWorldMatrix);
    Matrix4x4 projectionMatrix;
    photoCaptureFrame.TryGetProjectionMatrix(out projectionMatrix);


    var imagePosZeroToOne = new Vector2(pixelPos.x / imageWidth, 1 - (pixelPos.y / imageHeight));
    var imagePosProjected = (imagePosZeroToOne * 2) - new Vector2(1, 1);    // -1 to 1 space

    var cameraSpacePos = UnProjectVector(projectionMatrix, new Vector3(imagePosProjected.x, imagePosProjected.y, 1));
    var worldSpaceCameraPos = cameraToWorldMatrix.MultiplyPoint(Vector3.zero);     // camera location in world space
    var worldSpaceBoxPos = cameraToWorldMatrix.MultiplyPoint(cameraSpacePos);   // ray point in world space
 
    RaycastHit hit;
    bool hitToMap = Physics.Raycast(worldSpaceCameraPos, worldSpaceBoxPos - worldSpaceCameraPos, out hit, 20, SpatialMappingManager.Instance.LayerMask);
}

public static Vector3 UnProjectVector(Matrix4x4 proj, Vector3 to)
{
    Vector3 from = new Vector3(0, 0, 0);
    var axsX = proj.GetRow(0);
    var axsY = proj.GetRow(1);
    var axsZ = proj.GetRow(2);
    from.z = to.z / axsZ.z;
    from.y = (to.y - (from.z * axsY.z)) / axsY.y;
    from.x = (to.x - (from.z * axsX.z)) / axsX.x;
    return from;
}