EncodingError: Decoder error. (Unable to determine size of bitstream buffer.) when using Webcodec API

Question

I'm using Webcodecs since it now globally supported (except for mozilla mobile). My goal is to create a streaming plateform for cameras, I have a camera that sends it's datas to a server. This server has a websocket where i can connect my client. This websocket sends x264 NALUs and I want the client to decode this video and to display it on a canvas. To create the best UX, I choose to use it inside a worker.

This is the worker i made:

class Decoder {
    private decoder!: VideoDecoder;
    private sps!: Uint8Array;
    private pps!: Uint8Array;
    private isConfigured = false;
    private nalBuffer: Uint8Array[] = [];
    private canvas: OffscreenCanvas | null = null;

    set Canvas(canvas: OffscreenCanvas) {
        this.canvas = canvas;
    }

    constructor() {
        this.decoder = new VideoDecoder({
            output: (frame) => this.displayFrame(frame),
            error: (err) => {
                console.error('Decoder error:', err);
                // Attempt to reset decoder
                this.decoder.close();
                this.isConfigured = false;
                this.nalBuffer = [];
            }
        });
    }

    async decodeNAL(nal: Uint8Array) {
        if (this.decoder.state === "closed") return;

        // Suppression start code
        while (nal.length > 4 && nal[0] === 0x00) {
            nal = nal.slice(1);
        }
        nal = nal.slice(1);

        const nalType = nal[0] & 0x1F;

        switch (nalType) {
            case 7: // SPS
                if (!this.isConfigured) {
                    this.sps = nal;
                    this.nalBuffer.push(nal);
                }
                break;
            case 8: // PPS
                if (!this.isConfigured) {
                    this.pps = nal;
                    this.nalBuffer.push(nal);
                }
                break;
            case 5: // I-Frame
            case 1: // P-Frame
                if (!this.sps && !this.pps) {
                    console.warn("Skipping frame, waiting for configuration frames")

                    return;
                }

                if (!this.isConfigured && this.sps && this.pps) {
                    await this.configureDecoder();
                }
                this.processNal(nal);
                break;
        }
    }

    private processNal(nal: Uint8Array) {
        // Filter out unwanted NAL types if needed
        const acceptedNalTypes = [1, 5]; // Include more types if necessary
        if (!acceptedNalTypes.includes(nal[0] & 0x1F)) return;

        if (nal[0] & 0x1F) {
            this.nalBuffer = this.nalBuffer.slice(0, 3);
        } else {
            this.nalBuffer = this.nalBuffer.slice(0, 4);
        }

        this.nalBuffer.push(nal);

        this.decodeFrame();

    }

    private async configureDecoder() {
        try {
            const codecString = this.getAvcCodecString(this.sps);
            const description = this.convertToAvcC(this.sps, this.pps);

            const config: VideoDecoderConfig = {
                codec: codecString,
                description: description,
                hardwareAcceleration: "no-preference",
            };

            const isSupported = await VideoDecoder.isConfigSupported(config);

            if (!isSupported.supported) {
                console.error('Unsupported decoder config:', isSupported.config);
                throw new Error('No supported codec configuration found');
            }

            this.decoder.configure(config);
            this.isConfigured = true;
        } catch (error) {
            console.error('Decoder configuration failed:', error);
        }
    }

    private decodeFrame() {
        if (!this.isConfigured) {
            console.warn('Decoder not configured');
            return;
        }

        for (const nal of this.nalBuffer) {
            try {
                const chunkType = (nal[0] & 0x1F) === 5 ? 'key' : 'delta';

                const chunk = new EncodedVideoChunk({
                    type: chunkType,
                    data: nal,
                    timestamp: Date.now() * 1000
                });

                console.log(chunk.byteLength);

                this.decoder.decode(chunk);
            } catch (error) {
                console.error('Decoding error:', error);
            }
        }

        // Clear the buffer after processing
        this.nalBuffer = [];
    }

    private getAvcCodecString(sps: Uint8Array): string {
        if (sps.length < 4) return 'avc1.42E01E';

        const profile_idc = sps[1].toString(16).padStart(2, '0');
        const constraint_set = sps[2].toString(16).padStart(2, '0');
        const level_idc = sps[3].toString(16).padStart(2, '0');

        return `avc1.${profile_idc}${constraint_set}${level_idc}`;
    }

    private convertToAvcC(sps: Uint8Array, pps: Uint8Array): Uint8Array {
        const avcCLength = 12 + sps.length + pps.length;
        const avcC = new Uint8Array(avcCLength);

        avcC.set([1, sps[1], sps[2], sps[3], 0xff], 0);

        avcC.set([0xe1, (sps.length >> 8) & 0xff, sps.length & 0xff], 5);

        avcC.set(sps, 8);

        avcC.set([1, (pps.length >> 8) & 0xff, pps.length & 0xff], 9 + sps.length);

        avcC.set(pps, 12 + sps.length);

        return avcC;
    }

    private displayFrame(frame: VideoFrame) {
        if (!this.canvas) throw new Error('Canvas not configured');
        if ("getContext" in this.canvas) {
            // @ts-ignore
            this.canvas.getContext("2d").drawImage(frame, 0, 0, frame.displayWidth, frame.displayHeight);

        }

        frame.close();
    }
}

let decoder: Decoder = new Decoder();

self.addEventListener("message", async (e) => {
    switch (e.data.type) {
        case "buffer":
            await decoder.decodeNAL(new Uint8Array(e.data.data));
            break;
        case "canvas":
            decoder.Canvas = e.data.data as OffscreenCanvas;
    }
})

With this, I receive some I/P NALs that I ignore and then SPS and PPS NALs, with this NALs I configure the decoder using configureDecoder(). When this function ends, the decoder should be usable and create VideoFrame when running decode. But when I receive my first I frame after the decoder was configured I get this error: EncodingError: Decoder error. (Unable to determine size of bitstream buffer.). I don't really understand why this exception is thrown. I did not found any questions about this on the site, and no AI can help since it's niche and a little young

Answer 1

Thanks to this awnser I found that it was because I was using camera videos, this cameras uses annex B, which means i have a start code. AVCC do not need the start code but instead needs the size of the NAL.

Here is my method to convert Annex B to AVCC

    private ConvertAnnexB2AVCC(nal: Uint8Array): Uint8Array {
        const nalLength = nal.length;

        // Big endian size format
        const lengthHeader = new Uint8Array(4);
        lengthHeader[0] = (nalLength >> 24) & 0xFF;
        lengthHeader[1] = (nalLength >> 16) & 0xFF;
        lengthHeader[2] = (nalLength >> 8) & 0xFF;
        lengthHeader[3] = (nalLength) & 0xFF;

        // Create a new buffer with the size at the start
        let avccNal = new Uint8Array(4 + nalLength);
        avccNal.set(lengthHeader, 0);
        avccNal.set(nal, 4);

        return avccNal;
    }

Hope it'll help someone !