CODEWITHSUNDEEP
hololens
user1512979
user1512979

Reputation: 51

Access HoloLens depth image in Unity?

Does anyone know how to access the raw depth image from the Hololens depth camera in a Unity app? I know that I have to enable "research mode" on the Hololens. I have seen the example of "SensorStreamViewer" at https://github.com/Microsoft/HoloLensForCV/tree/master/Samples/SensorStreamViewer, but this is a DirectX app. I would like to use Unity because it seems easier than DirectX for prototyping and development (especially for a new developer like me!).

Incidentally, I plan to stream the depth images to an external desktop PC for object recognition, and receive the results back on the Hololens so that I can render holograms aligned with the real object.

Upvotes: 3

Views: 2396

Answers (3)

kühlschrankDino
kühlschrankDino

Reputation: 53

You can use it in Unity if you write a wrapper class and use HoloLensForCV as a winmd as in https://github.com/doughtmw/HoloLensForCV-Unity. And you can also convert the code in C# code. You need to convert the MediaFrameSourceGroup class. The stream data is accessible via the Guids.

Upvotes: 1

user1512979
user1512979

Reputation: 51

After talking to numerous people, including Microsoft tech support, I have found out that you can only access the Hololens depth camera image if you enable research mode and then write C++ code and use DirectX. It is not possible to access the depth camera using C# and Unity.

If someone else wants to pursue this, it may be possible to write C++ code that accesses the depth camera, and put that into a plugin dll that can be used from Unity. As an example, the following project shows how to access the webcam stream from within Unity (but not the depth camera): https://github.com/VulcanTechnologies/HoloLensCameraStream.

Upvotes: 2

Perazim
Perazim

Reputation: 1529

Dont know If I understood it right. I would recommend to check the SpatialMappingExample and SpatialProcessing. Its about visualizing and accessing spatial mapping data. Processing the raw spatial mapping data, finding and creating planes based on spatial mapping meshes... But right now they deleted all the guides because of the new MRTK V2 -.- so I cant give you reference

Even the references on the microsoft documentation (on the bottom) are referencing to a not exisiting page

Upvotes: 2

Related Questions