CODEWITHSUNDEEP
androidwebrtcwikitude
fraessig
fraessig

Reputation: 73

how to stream Webrtc's camera output to the Wikitude sdk on native android

In my webrtc app i am sending the captured VideoStream to a remote browser. My goal is now that i want to augment some AR Stuff with wikitude (i am using Wikitude 7.2.1 for compatibility reasons) on that same videostream on the android part and show it on the display (it currently only shows the camera output). I know that i can't just open another Camera Instance since Webrtc already uses one so i need to pass the output from the webrtc VideoCapturer (or alternatively the contents of some surface that output gets rendered too?) to the Wikitude SDK, however, i have difficulties doing so.

I have seen that the Input Plugin wikitude needs for an external Camera Input uses an ImageReader with an ImageAvaibleListener like this:

mWikitudeCamera2.start(new ImageReader.OnImageAvailableListener() { ... }

and on the WikitudeCamera2 side:

 mManager.openCamera(getCamera(), cameraStateCallback, null);
 mImageReader = ImageReader.newInstance(mFrameWidth, mFrameHeight, ImageFormat.YUV_420_888, 2);
 mImageReader.setOnImageAvailableListener(onImageAvailableListener, null);

that Image Readers surface then gets attached to the actual camera output:

CaptureRequest.Builder builder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD);
builder.addTarget(mImageReader.getSurface());

However since i need to use the provided CameraCapturer from Webrtc i am not able to do that.

Is there a way to get the camera stream from maybe the Webrtc's SurfaceViewRender, or render it to another (maybe fake?) SurfaceView and attach that to an Image reader. Or some other way to pass the output that gets sended to the client to wikitude without the use of an Image Reader? Or generally some other way i could get this to work that i havent thought off. I would really appreciate some help since im stuck on that part as of right now.

Upvotes: 1

Views: 333

Answers (1)

Andi F.
Andi F.

Reputation: 776

You're on the right path but unfortunately, since the Wikitude's SDK is consuming the camera API, it is not possible (yet, SDK 8.2) to access the same stream via the WebView / WebRTC. You have to stick to the ImageReader approach for now and find a way to process the frames in a native environment.

Best regards

Upvotes: 0

Related Questions