Extending a projection matrix

Question

I want to place a 3D heart inside of person using augmented reality. I have a projection matrix derived from an iPhone's camera intrinsics. Displaying objects relative to a framemarker works perfectly using the Vuforia SDKs generated projection matrix and pose. However, I would like to place an object in the scene behind the framemarker. To maintain the illusion any object behind the marker really needs to have the camera image in front of the object. Here is a drawing for the purposes of explanation.

Since the projection matrix will be the space in front of the image plane rather than the far plane, it seems to me I need to either extend the projection matrix or place the whole world inside another projection. Simply transposing the heart behind the marker doesn't work for the obvious reason you can never actually put anything behind the image plane. So the question I'm asking is, is this possible via some sort of projection matrix extension, or is there something else that would work?

Answer 1

Unity has something called DepthMask this basically allows an Object to appear behind the ImageTarget. But since your application requires you to see through the ImageTarget you could try within OnTrackingFound() and change ImageTarget's

• tranform.renderer.material.maintexture to an transparent material
• shader and implement an opacity decrease