How to create video srcObject from VideoFrame?

Question

I'm learning webcodecs now, and I saw things as below:

So I wonder maybe it can play video on video element with several pictures. I tried many times but it still can't work. I create videoFrame from pictures, and then use MediaStreamTrackGenerator to creates a media track. But the video appears black when call play().

Here is my code：

 const test = async () => {
    const imgSrcList = [
      'https://gw.alicdn.com/imgextra/i4/O1CN01CeTlwJ1Pji9Pu6KW6_!!6000000001877-2-tps-62-66.png',
      'https://gw.alicdn.com/imgextra/i3/O1CN01h7tWZr1ZiTEk1K02I_!!6000000003228-2-tps-62-66.png',
      'https://gw.alicdn.com/imgextra/i4/O1CN01CSwWiA1xflg5TnI9b_!!6000000006471-2-tps-62-66.png',
    ];
    const imgEleList: HTMLImageElement[] = [];

    await Promise.all(
      imgSrcList.map((src, index) => {
        return new Promise((resolve) => {
          let img = new Image();
          img.src = src;
          img.crossOrigin = 'anonymous';
          img.onload = () => {
            imgEleList[index] = img;
            resolve(true);
          };
        });
      }),
    );

    const trackGenerator = new MediaStreamTrackGenerator({ kind: 'video' });
    const writer = trackGenerator.writable.getWriter();
    await writer.ready;

    for (let i = 0; i < imgEleList.length; i++) {
      const frame = new VideoFrame(imgEleList[i], {
        duration: 500,
        timestamp: i * 500,
        alpha: 'keep',
      });
      await writer.write(frame);
      frame.close();
    }

    // Call ready again to ensure that all chunks are written before closing the writer.
    await writer.ready.then(() => {
      writer.close();
    });

    const stream = new MediaStream();
    stream.addTrack(trackGenerator);

    const videoEle = document.getElementById('video') as HTMLVideoElement;
    videoEle.onloadedmetadata = () => {
      videoEle.play();
    };
    videoEle.srcObject = stream;
  };

Thanks!

Answer 1

Disclaimer:
I am not an expert in this field and it's my first use of this API in this way. The specs and the current implementation don't seem to match, and it's very likely that things will change in the near future. So take this answer with all the salt you can, it is only backed by trial.

---

---

There are a few things that seems wrong in your implementation:

• duration and timestamp are set in micro-seconds, that's 1/1,000,000s. Your 500 duration is then only half a millisecond, that would be something like 2000FPS and your three images would get all displayed in 1.5ms. You will want to change that.
• In current Chrome's implementation, you need to specify the displayWidth and displayHeight members of the VideoFrameInit dictionary (though if I read the specs correctly that should have defaulted to the source image's width and height).

Then there is something I'm less sure about, but it seems that you can't batch-write many frames. It seems that the timestamp field is kind of useless in this case (even though it's required to be there, even with nonsensical values). Once again, specs have changed so it's hard to know if it's an implementation bug, or if it's supposed to work like that, but anyway it is how it is (unless I too missed something).

So to workaround that limitation you'll need to write periodically to the stream and append the frames when you want them to appear.

Here is one example of this, trying to keep it close to your own implementation by writing a new frame to the WritableStream when we want it to be presented.

const test = async() => {
  const imgSrcList = [
    'https://gw.alicdn.com/imgextra/i4/O1CN01CeTlwJ1Pji9Pu6KW6_!!6000000001877-2-tps-62-66.png',
    'https://gw.alicdn.com/imgextra/i3/O1CN01h7tWZr1ZiTEk1K02I_!!6000000003228-2-tps-62-66.png',
    'https://gw.alicdn.com/imgextra/i4/O1CN01CSwWiA1xflg5TnI9b_!!6000000006471-2-tps-62-66.png',
  ];
  // rewrote this part to use ImageBitmaps,
  // using HTMLImageElement works too
  // but it's less efficient
  const imgEleList = await Promise.all(
    imgSrcList.map((src) => fetch(src)
      .then(resp => resp.ok && resp.blob())
      .then(createImageBitmap)
    )
  );
  const trackGenerator = new MediaStreamTrackGenerator({
    kind: 'video'
  });

  const duration = 1000 * 1000; // in µs (1/1,000,000s)
  let i = 0;
  const presentFrame = async() => {
    i++;
    const writer = trackGenerator.writable.getWriter();
    const img = imgEleList[i % imgEleList.length];
    await writer.ready;
    const frame = new VideoFrame(img, {
      duration, // value doesn't mean much, but required
      timestamp: i * duration, // ditto
      alpha: 'keep',
      displayWidth: img.width * 2, // required
      displayHeight: img.height * 2, // required
    });
    await writer.write(frame);
    frame.close();
    await writer.ready;
    // unlock our Writable so we can write again at next frame
    writer.releaseLock();
    setTimeout(presentFrame, duration / 1000);
  }
  presentFrame();
  const stream = new MediaStream();
  stream.addTrack(trackGenerator);

  const videoEle = document.getElementById('video');
  videoEle.srcObject = stream;
};
test().catch(console.error)

<video id=video controls autoplay muted></video>