CODEWITHSUNDEEP
javascriptwebrtcvideo-streamingmedia
Shashank S
Shashank S

Reputation: 48

How to combine webcam and screen share video without using canvas?

I have a webrtc react app, where users can simulcast their stream to youtube, facebook etc (like restream.io).

I want to send both streams (screen share and webcam) as one video (half screen share and half webcam, webcam overlayed on screen share, captions on top of video) like studio.restream.io

Everything is working fine with drawing streams on canvas and piping data using websocket to backend where it is transcoded to rtmp and sent to fb,yt etc.(This method is working only in high end PC).

But the only problem with this method is when i draw stream on canvas it takes lot of cpu and browser hangs(only works when you have gpu).

The question is how to optimize this?

Do we need a back-end service to merge videos using ffmpeg? or Is there any way to do it in browser?

Upvotes: 1

Views: 940

Answers (1)

Kwindla Kramer
Kwindla Kramer

Reputation: 141

In general, the canvas operations (and a lot of other drawing-related operations) in the browser assume a GPU is available and are very slow when they have to run on the CPU.

For what you're doing, you probably do need to run the browser on hardware that has a GPU.

You're right that you can do this kind of compositing more flexibly using ffmpeg or GStreamer. We've used both ffmpeg and GStreamer pretty extensively, at Daily.co.

For our production live streaming workers, we use GStreamer running on AWS instances without GPUs. Our media servers forward the WebRTC rtp tracks as raw rtp to a GStreamer process, which decodes the tracks, composites the video tracks, mixes the audio tracks, and encodes to RTMP. GStreamer has a steep learning curve and is a totally different toolkit than the browser, but it's also efficient and flexible in ways that running in the browser can't quite be.

Upvotes: 4

Related Questions