CODEWITHSUNDEEP
ioswebrtcvideo-capturecvpixelbuffer
Giraff Wombat
Giraff Wombat

Reputation: 118

WebRTC iOS: Filtering camera stream from RTCCameraVideoCapturer. Conversion from RTCFrame to CVPixelBuffer

I found the git below is simple and efficient by using func capturer(_ capturer: RTCVideoCapturer, didCapture frame: RTCVideoFrame) of RTCVideoCapturerDelegate. You get RTCVideoFrame and then convert to CVPixelBuffer to modify.

https://gist.github.com/lyokato/d041f16b94c84753b5e877211874c6fc

However, I found Chronium says nativeHandle to get PixelBuffer is no more available(link below). I tried frame.buffer.pixelbuffer..., but, looking at framework > Headers > RTCVideoFrameBuffer.h, I found CVPixelBuffer is also gone from here!

https://codereview.webrtc.org/2990253002

Is there any good way to convert RTCVideoFrame to CVPixelBuffer? Or do we have better way to modify captured video from RTCCameraVideoCapturer?

Below link suggests modifying sdk directly but hopefully we can achieve this on Xcode.

How to modify (add filters to) the camera stream that WebRTC is sending to other peers/server

Upvotes: 2

Views: 3031

Answers (1)

Sumit Meena
Sumit Meena

Reputation: 529

can you specify what is your expectation? because you can get pixel buffer from RTCVideoframe easily but I feel there can be a better solution if you want to filter video buffer than sent to Webrtc, you should work with RTCVideoSource.

you can get buffer with as seen

    RTCCVPixelBuffer *buffer = (RTCCVPixelBuffer *)frame.buffer;
    CVPixelBufferRef imageBuffer = buffer.pixelBuffer;

(with latest SDK and with local video camera buffer only)

but in the sample i can see that filter will not work for remote. enter image description here

i have attached the screenshot this is how you can check the preview as well.

Upvotes: 6

Related Questions