CODEWITHSUNDEEP
unity-game-engineroshololensmrtk
12-ko
12-ko

Reputation: 1

Communicate with Hololens2 and Ros using rossharp, and publish the camera image of Hololens2 as raw_image

I started application development for Hololens2 lately. And I'd like to using rossharp between Unity and Ros.

Now, I'm trying to publish the camera image of Hololens2 to ros within the app launched by Hololens2. (Introducing MRTK as a tool.)

First, I attached the ImagePublisher script into Gameobject, and then selected MixedRealityPlaySpace / Main Camera from Hierarchy for the ImageCamera. However, when I play it, I got an error that Display1 No Cameras Rendering.

So when I duplicated the Main Camera, it went well to Build, the app worked well, and the data was sent. Although, when I visualized this image using rqt_image_view, the camera image of Hololens 2 is not displayed well because the entire screen is gray.

I searched a lot, but I couldn't find a place using this script in the first place, and I was at a loss without a model.

I tried the following:

I'm still new to these things and it's difficult to improve on my own. So I want to get someone's help. Thanks.

Upvotes: 0

Views: 842

Answers (3)

12-ko
12-ko

Reputation: 1

I realized the request by using the Capture a photo to a Texture2D with location of PhotoCapture function and Ros-sharp.

Concretely, after capture a photo to Texture2D with PhotoCapture function, substitute the Texture2D for variable in ImagePublisher script. And I succeeded in receiving by decompressing the data delivered by the above method.

Upvotes: 0

Shivani Guptasarma
Shivani Guptasarma

Reputation: 1

In my current understanding (which could be wrong, as I'm also new to this), a Unity camera is like a point-of-view that displays a scene to the user within the game/app, rather than a physical camera facing the outside world. So, assuming you wish to access the front RGB camera of the Hololens and get real-world images, the ImagePublisher script doesn't seem to know how to access it.

Here is a nice video by Peter Koch that shows how to use the WebCamTexture class to get data from a hardware camera. I put together a modified script using Peter's code to get camera images, and the original ROS# ImagePublisher code to publish them into ROS. I'm not sure if this was necessary or could have been done in a better way, but I'm able to see pixel data in the terminal as well as the video stream in rqt_image_view after deploying to Hololens 2. You're welcome to download it or have a look, if it's of use to you: https://github.com/armlabstanford/mixed_reality_setup#accessing-the-hololens-front-camera-and-streaming-camera-data-to-ros.

Upvotes: 0

Cornelius Einem
Cornelius Einem

Reputation: 60

It has been a long time since I worked with ROS and the HoloLens, but back then I had more success in working with ROS2-dotnet rather than rossharp, in case you want to give it a shot.

Upvotes: 0

Related Questions