CODEWITHSUNDEEP
unity-game-enginehololens
Eric Smith
Eric Smith

Reputation: 186

Object Recognition with Mixed Reality Capture (MRC)

We're using the HoloLens' locatable camera (in Unity) to perform a number of image recognition tasks. We'd like to utilize the mixed reality capture feature (MRC) available in the HoloLens developer portal so that we can demo our app, but MRC crashes because we're hogging the camera in Photo Mode.

Does anyone have a good workaround for this? We've had some ideas, but none of them are without large downsides.

Solution: Put your locatable camera in Video Mode so that you can share the video camera with MRC. Downside: Video Mode only allows us to save the video to disk, but we need realtime access to the buffer in memory (the way photo mode gives us access) so that we can do our detection in realtime.

Solution: Capture the video in a C++ plugin, and pass the frame bytes to Unity. This allows MRC to work as expected. Downside: We lose the 'locatable' part of the 'locatable camera' as we no longer get access to the cameraSpaceToWorldSpace transformation matrix, which we are utilizing in our UI to locate our recognized objects in world space. Sub-solution: recreate the locatable camera view's transformation matrix yourself. Sub-downside: I don't have any insight into how Microsoft creates this transformation matrix. I imagine it involves some hardware complexities, such as accounting for lens distortions. If someone can guide me to how this matrix is created, that might be one solution.

Solution: Turn off object recognition while you create the MRC, then turn it back on when you're done recording Downside: Our recognition system runs in real time, n times per second. There would be no way to capture the recognitions on video.

Upvotes: 1

Views: 722

Answers (1)

Eric Smith
Eric Smith

Reputation: 186

We ended up creating a plugin for Unity that uses Microsoft's Media Foundation to get access to the video camera frames. We open sourced it in case anyone else runs into this problem.

The plugin mimics Unity's VideoCapture class so that developers will be able to easily understand how to implement it.

Hopefully this is helpful to a few.

Upvotes: 2

Related Questions