CODEWITHSUNDEEP
videorustvideo-streaminggstreamervideo-processing
Ferdia McKeogh
Ferdia McKeogh

Reputation: 451

Lowest overhead source format for GStreamer?

I need to generate a live overlay in Rust for use as a GStreamer source.

I have a few mutable variables; an array of strings and some sensor data that need to be overlayed on a 1080p60 video stream.

Is generating a raw stream best, or encoding to MJPG first? What method has the least total overhead between the Rust application and GStreamer?

Upvotes: 1

Views: 425

Answers (1)

Sebastian Dröge
Sebastian Dröge

Reputation: 2143

There are multiple options here for you, depending on how you want to generate the overlay. I wouldn't recommend going via MJPEG or anything else but just work on raw video. This is quite convenient with the Rust GStreamer bindings and has basically the lowest possible overhead.

1) Generate a overlay raw video stream, each frame being fully transparent where there is no overlay. This can be done similar to the appsrc example from the Rust GStreamer bindings here

You could then pass that to the compositor element (or glvideomixer or other mixer elements using some hardware API), which could composite your normal video stream and the overlay stream.

For generating the overlay you could use any API you want that can render text, etc on top of ARGB (or any other raw video format). One option would be Cairo, which also has Rust bindings.

It has the disadvantage over the next solution that you have to generate full video frames, the compositor has to generate yet another video frame for each frame. A lot of copying will have to happen.

2) You could write a GStreamer element in Rust does acts as a video filter and paints the overlay over the video frames. There is an example video filter here and I wrote an article explaining each part here

You don't have to do all the plugin initialization stuff in src/lib.rs but could have only the equivalent of the source file of the RGB2Gray converter inside your application and call the register function somewhere at the beginning of your code.

The video filter itself would also be a bit simpler as input and output format would be the same.

For generating the overlay you could use any API you want that can render text, etc on top of ARGB (or any other raw video format). One option would be Cairo, which also has Rust bindings.

Apart from potentially needed video format conversion (if the input video has a different format than what you support for drawing on top of it), this would be the most efficient approach.

3) Make use of the cairooverlay GStreamer element. It basically works like 2) and gives you directly a Cairo context that you can then use for drawing whatever you want on top of the video frame. You get a signal with a Cairo context (the "draw" signal) for every frame that passes through the element.

But you have to use the Cairo API then, and you might have to add some API extensions to the Rust Cairo bindings. I'm not 100% sure if everything needed is already available.

Upvotes: 3

Related Questions