IoT-Practitioner
IoT-Practitioner

Reputation: 81

What is the the most efficient way of endering an MJPEG-Stream in browser

My application displays groups of MP4 (.H264) camera streams (displayed side by side) very efficiently. This is utilising feeding fragmented MP4 segments to a video element.

I am currently adding MJPEG support for legacy ip cameras, that can not provide .H264. This is however, as expected, quite expensive in the browser. I am looking for the absolutely most efficient way to implement this functionality.

My current test method:

Backend (Node.js, in a spawned child process) I am pulling an MJPEG-RTSP stream with FFMPEG, muting all channels(data/subtitels, audio) but the MJPEG stream itself.

The output is piped to sockets io thus:

    const IOptions = {
        path: '/streams/' + cameraID
    }

And

        Spawned.stdio[x].on('data', function (data) {

            let frame = Buffer.from(data).toString('base64')

            socket.emit('canvas', frame)
        })

Frontend I am currently creating new image Objects to be drawn to canvas:

let socket = io('/', { path: '/streams/camtitle' })
socket.on('canvas', function (data) {
    try {
        const canvas = document.getElementById('mjpegCanvas')
        const context = canvas.getContext('2d')
        const imageObj = new Image()
        imageObj.src = "data:image/jpeg;base64," + data
        imageObj.onload = function () {
            context.height = imageObj.height
            context.width = imageObj.width
            context.drawImage(imageObj, 0, 0, context.width, context.height)
        }
    } catch (e) { console.log('Canvas Drawing Error:' + e) }
})

This is leads to varying processer core loads between 8% and 29% client side on VGA (640 x 480, low framerates) and users would expect to be able to feed in much higher resolutions and framerates.

I have delved into related topics over the last 5 days and found senior sources stating that the generation of Objects is generally a very costly process in Javascript, possibly also related to garbage collection.

Is there a better way of handling or even reusing my image Objects? As far as my understanding goes, I am reusing my const imageObj. Is there more happening in the background?

I have seen other cases on stackoverflow, who implement MJPEG via image tags, with the source pointing to a MJPEG stream like this:

<img src="http://[some ip]:[port]/mjpg"> 

Others simply point the image source to a route ending with .jpg.

Either seems to however not work with the MJPEG-Sream I am providing. It also surpasses my understanding how this should actually work, without passing JPGs or at least splitting data by the respective frame delimiter. What am I missing here? I would like to benchmark that method for comparison.

I am additionally implementing a test diffing function server side, that reduces the work by only sending frames with a certain measure of pixel changes. It will be interesting to see how costly that will turn out to be.

Any optimisation suggestions or highly applicable resources pointing me into the right direction would be highly appreciated. Vanilla Javascript without libraries and moduls are preferred, but other options will be considered as well.

The solution can also suggest changes on the server. This is all just experimental at this point.

Upvotes: 0

Views: 38

Answers (0)

Related Questions