WebCodecs - create mp4 video from jpg images

Question

I started playing with WebCodecs API. I try to create mp4 video from set of jpg images. Starting with something simple, I'm trying to create an mp4 from one jpg image. When the output callback is invoked it is saving video chunk into the mp4 file.

 const downloadVideo = (data) => {

       console.log("downlaodVideo data:" + data);
       const link = document.createElement("a");
 
       const file = new Blob([ Uint8Array.from( data ) ] , {type: "application/octet-stream"});
       link.href = URL.createObjectURL(file);
       link.download = "video.h264";
       link.click();
       URL.revokeObjectURL(link.href);
  };

$(document).ready(async function () {


     const videoEncoder = new VideoEncoder(
                {
                async output(chunk, metadata) {
                    console.log("encoder queue:" + videoEncoder.encodeQueueSize);
                    console.log("timestamp:" + chunk.timestamp + " counter:" + counter);
                    console.log("length:" + chunk.byteLength);
                    console.log("video chunk type:" + chunk.type);
                    console.log("duration:" + chunk.duration)
                    console.log(JSON.stringify(metadata));
                    console.log("decoder config:" + metadata.decoderConfig );
                    
                    var myMetadata;
                    var videoBlob;

                    if (metadata.decoderConfig) {
                        //# save the decoder's description/config ...
                        //# is SPS and PPS metadata needed by players to display H.264)
                        console.log("decoder config description:" + metadata.decoderConfig.description);
                        myMetadata = new Uint8Array( metadata.decoderconfig.description );
                    }

                    videoBlob = new ArrayBuffer(chunk.byteLength);
                    chunk.copyTo(videoBlob);

                    //# combine the two arrays (in the shown order of appearance)
                    var outputBytes = [...myMetadata, ...videoBlob];
  

                    downloadVideo(outputBytes);
                },
                error(error) {
                    console.log(output_Bytes);
                },
            }
            );
    

            const encoderConfig = {
                codec: "avc1.42001E",
                avc: { format: 'annexb' },
                height: 480,
                width: 640,
                framerate: 1,
                latencyMode: "realtime",
                bitrate: 2_000_000, // 2 Mbps
            };



            const support = await VideoEncoder.isConfigSupported(encoderConfig);
                
            if (support.supported) {
                console.log("Video Encoder configured!");
                videoEncoder.configure(encoderConfig );
            }
            else {
                console.error("Video codec not supported!");
            }




            $('#create_movie').click( function(ev) {

                const myImage = new Image();
                myImage.src = "samples/pic0.jpg";

                var imageBitmap = null;
                var imageBitmapPromise = null;

                myImage.onload = () => {

                    imgContext.drawImage(myImage,0,0);
                    imageBitmapPromise = createImageBitmap(myImage);

                    imageBitmap = imageBitmapPromise.then( result => {
                        const ms = 1_000_000; // 1µs
                        const fps = 10;
                        return new VideoFrame(result, {
                            timestamp: (ms * 1) / fps,
                            duration: ms / fps
                        },false);
                    }).then(vidFrame => {
                        console.log("imageBitmap:" + vidFrame);
                        console.log("video frame timestamp:" + vidFrame.timestamp);
                        console.log("format:" + vidFrame.format);
                        videoEncoder.encode(vidFrame, { keyFrame: true });

                        vidFrame.close();
                    });

                }


            });

}

Unfortunately, the video opens with an error message: The file you trying to play is empty file I generated mp4 from the same jpg file with ffmpeg tool:

ffmpeg  -i pic%d.jpg -vcodec mpeg4 test.mp4

The structure of both files is quite different:

On the left is video generated by ffmpeg. It contains some metadata that are missing in vido file generated by my code. So my question is: how to properly generate video file using WebCodecs API?

Answer 1

"I try to create mp4 video from set of jpg images"

WebCodecs doesn't do any muxing into containers like MP4 or AVI etc. It only gives the raw encoded frames (video or audio) and then the code-writer themselves will decide on an output container format (if needed).

Are you assuming you need MP4 to display it?
If yes: H.264 is playable as it is, since it's a video format.

Or you know for sure that you need MP4 output (eg: for HTML5 playback)?
If yes: Good luck. Not hard but very tedious to double-check everything.

Is it for preview purposes?

Use a player like VLC to play your saved H.264 file If you simply want to check that your code worked fine. VLC runs on desktop (so just drag your file into its window to check the visuals).

For online playback you can either find a JS-ready muxer like: JMuxer as shown in the code below, or else enjoy learning about MP4 structure then write the code. For writing own MP4 bytes I tried to explain a starting point on another Question, but they never responded.

Outside of webcodecs, you can just draw images into canvas then encode into MP4 using H.264-MP4-Encoder.

You first need a correctly encoded H.264 file or no player will accept it.
(see below for fixes to your code)...

To encode a playable raw H.264:

You need to set Annex-B as the output format

videoEncoder.configure( {
                codec: "avc1.42001E",
                avc: { format: 'annexb' },
                height: 480,
                width: 640,
                framerate: 1,
                bitrate: 2_000_000, // 2 Mbps
            });

To encode your H.264 contained inside MP4 (using JMuxer):

<!DOCTYPE html>
<html>
<body>

<button id="create_movie" >  create video </button>

<br><br>

<!-- for preview of input Image's pixels -->
<canvas id="myCanvas" width="320" height="240" > </canvas

<!-- for preview of JMuxer API result -->
<video id="player" width="640" height="480" > </video>

<!-- for using JMuxer API -->
<script src="https://cdn.jsdelivr.net/npm/jmuxer@2.0.5/dist/jmuxer.min.js"> 
</script>

<script>

const player = document.getElementById("player");
const canvas = document.getElementById("myCanvas");
var ctx = canvas.getContext("2d");
            
//# run function create_movie() on click ...
document.getElementById("create_movie").addEventListener( "click", create_movie );

//var myImage = new Image();
var myImage = document.createElement( "img");
var imageBitmap; var imageBitmapPromise;
var videoBlob;


const encoder_init =    {
                            output: handle_chunk,
                            error: (e) => { console.log("::: Webcodecs Encoder Error ::: " + "/n" + e.message) },
                        };

const encoder_config =  {
                            //codec: "avc1.42001E",
                            codec: "avc1.42C016",
                            avc: {  format: "annexb" },
                            //avc: {  format: "avc" },
                            width: 640,
                            height: 480,
                            framerate: 1,
                            latencyMode: "realtime",
                            bitrateMode: "constant",
                            bitrate: 2_000_000, // 2 Mbps
                        };

//# will update entry values as needed per frame encoding
const chunk_init =  {
                        type: "null",
                        timestamp: 0,
                        duration: 5000,
                    };
                    


const videoEncoder = new VideoEncoder( encoder_init );
videoEncoder.configure( encoder_config );

console.log("video encoder created! " + videoEncoder);

//# if using JMuxer to create MP4 container 
var jmuxer = new JMuxer(
                            {
                                node: "player", //# ID of <video> tag for showing output
                                mode: "video",
                                flushingTime: 100,
                                fps: 1,
                                maxDelay: 1,
                                clearBuffer: true,
                                debug: true
                            }
                        );

function create_movie( )
{
    //alert("create_movie");
    
    //# start loading an Image as the input frame
    load_image_frame("test_640x480.jpg");
    
}

async function load_image_frame( input_path )
{
    myImage.addEventListener("load", () => { handle_load_image(); } );
    
    myImage.src = input_path;
    myImage.decode();
}

async function handle_load_image( )
{
    let img_BMP = await createImageBitmap(  myImage );
    //alert( img_BMP );

    //# (optional) preview pixels on canvas
    ctx.drawImage(img_BMP, 0, 0, 640, 480, 0, 0, 320, 240);
    
    //# func encode_frame params --> ( in_Bitmap , in_type , in_timestamp , in_duration )
    encode_frame( img_BMP, "key" , 0 , 5000 );
}

function encode_frame( in_Bitmap , in_type , in_timestamp , in_duration )
{
    //alert("encode_frame");
    
    //# per frame settings...
    chunk_init.type = in_type;
    chunk_init.timestamp = in_timestamp;
    chunk_init.duration = in_duration;
    
    curr_Frame = new VideoFrame( in_Bitmap, chunk_init );
    
    if (videoEncoder.encodeQueueSize > 2) 
    {
        //# drop this frame (since 2 is too many frames in 1 queue)
        curr_Frame.close();
    } 
    else 
    {
        //# encode as "keyframe" (or false == a "delta" frame)
        videoEncoder.encode( curr_Frame , { keyFrame: true } );
    }   
}

async function handle_chunk( chunk, metadata ) 
{
    console.log("timestamp: " + chunk.timestamp);
    console.log("length: " + chunk.byteLength);
    console.log("video chunk type: " + chunk.type);
    console.log("duration: " + chunk.duration)
    
    //console.log(JSON.stringify(metadata));
    
    //# metadata shows only if the format is "avc" (eg: is not needed for "annexb") ....
    if( encoder_config.avc.format == "avc" )
    {
        if (metadata.decoderConfig) 
        {
            //# save the decoder description ( is SPS and PPS for AVC/MP4 )
            myConfigBytes = new Uint8Array(  metadata.decoderConfig.description );
            
            //# preview the description... (is bytes Integers, not String chars)
            alert( 
                    "Config bytes for MP4 \"stsd\" atom : " + "\n" +
                    print_hex_from_arr( myConfigBytes ) 
                 );
        }
        
    }

    //# get actual bytes of H.264 encoded data ...
    videoBlob = new Uint8Array( chunk.byteLength );
    chunk.copyTo( videoBlob );
    
    //### choose output option: raw H264 file or MP4 created with JMuxer
    
    //### option 1 : H.264 --> save playable raw H.264 file
    //downloadVideo(videoBlob);
    
    //### option 2 : MP4 --> encode MP4 using JMuxer
    encode_MP4( videoBlob );
    
    //# clear encoder buffers
    videoEncoder.flush();
    
}

function encode_MP4( videoBlob )
{
    //# show play controls (if wanted)
    player.controls = "controls";
    
    //# add an encoded video frame for streaming
    jmuxer.feed(
                    {
                        video: videoBlob,
                        duration: 5000
                    }
                );  
}

function print_hex_from_arr( input_array )
{
    let tmp_hex_str = "";
    let tmp_byte_val = "";
    
    //# write each value into an array slot
    for ( let i=0; i < input_array.length; i++ ) 
    { 
        tmp_byte_val = input_array[i].toString(16);
        
        if( (tmp_byte_val.length % 2) !== 0 ) 
        { tmp_byte_val = ( "0" + tmp_byte_val ); }
        
        tmp_hex_str += tmp_byte_val;
        tmp_hex_str += " ";
    }
    
    return ( tmp_hex_str.toUpperCase() );
}


</script>

</body>
</html>