CODEWITHSUNDEEP
javascripthtmlhtml5-videomediarecordermedia-source
stevendesu
stevendesu

Reputation: 16831

HTML5 DVR not working -- SourceBuffer removed from parent element

Goal

I'm attempting to create a rudimentary "DVR" for an HTML5 video element by utilizing MediaRecorder, MediaSource, and SourceBuffer. At the moment this is just a proof of concept. However since many projects like HLS.js take advantage of the HTML5 video element, I believe this would have wide-spread value.

Code

Here's the gist of my code:

<html>
<head>
</head>
<body>
    <video id="src-video" src="http://localhost:8080/video/source.mp4" autoplay></video>
    <video id="dvr-video"></video>
    <input id="seekbar" type="range" min="-120" max="0" value="0" />
    <script>
    var mr; // MediaRecorder
    var ms = new MediaSource();
    var srcBuf; // SourceBuffer
    var srcUrl = URL.createObjectURL(ms);
    var srcVid = document.getElementById("src-video");
    var dvrVid = document.getElementById("dvr-video");
    var dvrData = []; // array of ArrayBuffer
    var queue = [];

    ms.addEventListener("sourceopen", sourceOpen);
    srcVid.addEventListener("playing", setupMediaRecorder);
    dvrVid.src = srcUrl;

    var seekBar = document.getElementById("seekbar");
    seekBar.addEventListener("change", function(e) {
        // Destroy the old media source and make a new one
        URL.revokeObjectURL(srcUrl);
        srcBuf = null;

        ms = new MediaSource();
        ms.addEventListener("sourceopen", sourceOpen);
        srcUrl = URL.createObjectURL(ms);

        body.removeChild(dvr);
        dvr = document.createElement("video");
        body.insertBefore(dvr, seekBar);

        dvr.src = srcUrl;
    });

    function sourceOpen()
    {
        // Create the source buffer
        if (!srcBuf)
        {
            srcBuf = src.addSourceBuffer('video/webm; codecs="opus,vp8"');
            srcBuf.mode = "sequence";
        }

        srcBuf.addEventListener('updateend', function() {
            if ( queue.length ) {
                srcBuf.appendBuffer(queue.shift());
            } else {
                dvr.play();
            }
        }, false);

        // Add all fragments in cache
        var start = dvrData.length + parseInt(seekBar.value);
        queue = [];
        for( var i = start; i < dvrData.length; i++ )
        {
            if (dvrData[i])
                queue.push(dvrData[i]);
        }
        if (queue.length)
            srcBuf.appendBuffer(queue.shift());
    }

    function setupMediaRecorder()
    {
        var stream = srcVid.captureStream()
        mr = new MediaRecorder(stream);
        mr.ondataavailable = function(e) {
            // Convert the Blob to an ArrayBuffer
            var fileReader = new FileReader();
            fileReader.onload = function() {
                // Append this ArrayBuffer to our playing video
                if (srcBuf)
                {
                    if (srcBuf.updating || queue.length)
                        queue.push(this.result);
                    else
                        srcBuf.appendBuffer(this.result);
                }
                // And to our historical array (for seeking purposes)
                dvrData.push(this.result);
                if (dvrData.length > 120) {
                    // Keep only 2 minutes of data
                    dvrData.splice(0, 1);
                }
            };
            fileReader.readAsArrayBuffer(e.data);
        };
        mr.start();
        // Record 1-second chunks
        setInterval(function() {
            mr.requestData();
        }, 1000);
    }
    </script>
</body>
</html>

Result

When the page first loads, the "live" video element begins to play, and 1 second later the "dvr" element begins to play - with a 1-second delay. So it seems to be working at first.

The moment I perform a seek, the dvr element goes black and I get the following error in the console (line number may not match exactly with code above):

Uncaught DOMException: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
at SourceBuffer.<anonymous> (http://localhost:8080/video/dvr.html:80:13)

Looking at chrome://media-internals for further details, I see the following for the DVR player:

+-------------+----------------+--------------------------------------------------------------------------------------------
| Timestamp   | Property       | Value
+-------------+----------------+--------------------------------------------------------------------------------------------
| 00:00:00 00 | origin_url     | http://localhost:8080/
| 00:00:00 00 | frame_url      | http://localhost:8080/video/dvr.html
| 00:00:00 00 | frame_title    |
| 00:00:00 00 | url            | blob:http://localhost:8080/cec4134a-4498-43c5-8321-3743761636ac
| 00:00:00 00 | info           | ChunkDemuxer: buffering by DTS
| 00:00:00 00 | pipeline_state | kStarting
| 00:00:00 03 | error          | Unexpected element ID 0xa3
| 00:00:00 03 | error          | Append: stream parsing failed. Data size=112300 append_window_start=0 append_window_end=inf
| 00:00:00 08 | pipeline_error | CHUNK_DEMUXER_ERROR_APPEND_FAILED
| 00:00:00 10 | pipeline_state | kStopping
| 00:00:00 10 | pipeline_state | kStopped
+-------------+----------------+--------------------------------------------------------------------------------------------

The Unexpected element ID 0xa3 seems to be the culprit. Although for some reason this error wasn't thrown when the page first loaded (I'm appending the same ArrayBuffers to my SourceBuffer, so if they didn't throw this error before I don't know why they're throwing it now)

Looking up 0xa3 with respect to the WEBM format, it sounds like this refers to a "SimpleBlock" -- https://chromium.googlesource.com/webm/libwebm/+/libwebm-1.0.0.26/webmids.hpp -- I don't know why this would throw an error?

Things I've tried

So far I haven't had any luck getting DVR to work correctly. What am I missing?

Upvotes: 4

Views: 1841

Answers (1)

stevendesu
stevendesu

Reputation: 16831

I've got it

After a lot more experimentation I finally figured out the issue.

WEBM files are effectively binary-encoded XML files. The schema looks something like this:

<EBML>
    <EBMLVersion>...</EBMLVersion>
    <EBMLReadVersion>...</EBMLReadVersion>
    <EBMLMaxIDLength>...</EBMLMaxIDLength>
    <EBMLMaxSizeLength>...</EMBLMaxSizeLength>
    <DocType>...</DocType>
    <DocTypeVersion>...</DocTypeVersion>
    <DocTypeReadVersion>...</DocTypeReadVersion>
</EBML>
<Segment>
    <SeekHead>
        <Seek>...</Seek>
        <Seek>...</Seek>
        <Seek>...</Seek>
    </SeekHead>
    <Void></Void>
    <Info>...</Info>
    <Tracks>
        <TrackEntry>
            <Video>
                <Colour>
                    <MatrixCoeffciient>...</MatrixCoefficients>
                    ...
                </Colour>
            </Video>
            <Audio>
                ...
            </Audio>
        </TrackEntry>
    </Tracks>
    <Cues>
        <CuePoint>
            <CueTrackPositions>...</CueTrackPositions>
        </CuePoint>
    </Cues>
    <Cluster>
        <Timecode>...</Timecode>
        <SimpleBlock>...</SimpleBlock>
        <SimpleBlock>...</SimpleBlock>
        <SimpleBlock>...</SimpleBlock>
        <SimpleBlock>...</SimpleBlock>
        ...
    </Cluster>
    <Cluster>
        <Timecode>...</Timecode>
        <SimpleBlock>...</SimpleBlock>
        <SimpleBlock>...</SimpleBlock>
        <SimpleBlock>...</SimpleBlock>
        <SimpleBlock>...</SimpleBlock>
        ...
    </Cluster>
    ...
</Segment>

The way I was reading the data, the first block contained all of the header information (EBML data, segment data, tracks, timecode, etc) and all later chunks were just a stream of <Cluster> and <SimpleBlock> tags (with an odd <Timecode> every now and then)

Ultimately what I had to do was build a rudimentary demuxer to parse through the EBML file and extract the header information. Then whenever I performed a seek, this header information was injected into the buffer before any video data.

My Advice

Don't bother. MSE is terrible and this was a 72 hour nightmare. Save yourself the headache.

Upvotes: 3

Related Questions