CODEWITHSUNDEEP
typescriptwebcodecs
FrostKiwi
FrostKiwi

Reputation: 821

How to playback containerless / raw .h264 streams via Webcodecs API?

I want to play back a raw .h264 stream using the Webcodecs API and extract specific frames. It is saved as a .h264 file, essentially just the video data before being muxed into a .mp4 container format.

It should be possible to play this back using the new WebCodecs API. I fail at providing the avc1.64001e VideoDecoder with the properly formatted EncodedVideoChunk. How can I do so? I know the resolution, framerate and frame count, but how do I take this information and create an EncodedVideoChunk from that?

I have a potentially gigabytes big local file, which I receive via:

        const readableStream = this.data.file.stream();
        const reader = readableStream.getReader();

How to feed this into VideoDecoder: decode()? I know you can let mp4box.js handle the demuxing and providing of chunks. But I have the data already demuxed. So how do I overcome this step, besides muxing and feeding it through mp4box.js, only to have it demux again?

Upvotes: 1

Views: 821

Answers (1)

mpromonet
mpromonet

Reputation: 11963

In order to easily decode it is convenient to split stream grouping SPS + PPS + IDR, this could be done with :

for (let i = 0; i < buffer.length; i++) {
    if (buffer[i] === 0 && buffer[i+1] === 0 && buffer[i+2] === 0 && buffer[i+3] === 1 && (buffer[i+4]&0x1f !== 7) && (buffer[i+4]&0x1f) !== 8) {
        if (i !== start) {
            // process a frame
        }
        start = i;
    }
}

A simple decoding, with minimal error case management, setting type and codec to dummy values, using annexb will let VideoDecoder to overide values with frame content, could be :

async function process(data) {
    if (decoder.state !== "configured") {
        await decoder.configure({ codec: "avc1.64001e" });  
    }
    const chunk = new EncodedVideoChunk({
        timestamp: (performance.now() + performance.timeOrigin) * 1000,
        type: "key",
        data,
    });
    return decoder.decode(chunk);
}

Putting all together:

const decoder = new VideoDecoder({
    output: (frame) => {
        canvas.getContext("2d").drawImage(frame, 0, 0, frame.displayWidth, frame.displayHeight);
        frame.close();
    },
    error: (e) => console.warn(e.message),
})

async function process(data) {
    if (decoder.state !== "configured") {
        const config = { codec: "avc1.4d002a" };
        await decoder.configure(config);  
    }
    const chunk = new EncodedVideoChunk({
        timestamp: (performance.now() + performance.timeOrigin) * 1000,
        type: "key",
        data,
    });
    return decoder.decode(chunk);
}

function load() {
    const file = document.getElementById("file").files[0];
    const stream = file.stream();
    const reader = stream.getReader();
    let buffer = new Uint8Array();
    return reader.read().then(async function processChunk({done, value}) {
        if (done) {
            return process(buffer);
        } else {
            buffer = new Uint8Array([...buffer, ...value]);
            let start = 0;
            for (let i = 0; i < buffer.length; i++) {
                if (buffer[i] === 0 && buffer[i+1] === 0 && buffer[i+2] === 0 && buffer[i+3] === 1 && (buffer[i+4]&0x1f !== 7) && (buffer[i+4]&0x1f) !== 8) {
                    if (i !== start) {
                        const frame = buffer.slice(start, i);
                        await process(frame);
                    }
                    start = i;
                }
            }
            buffer = buffer.slice(start);
            return reader.read().then(processChunk);
        }
    });
}
<input type="file" id="file" onchange="load()" />
<canvas id="canvas"></canvas>

Upvotes: 1

Related Questions