CODEWITHSUNDEEP
javascripthtml5-videowebcodecsmediastreamtrack
kevind
kevind

Reputation: 56

How to process each frames exactly only once using MediaStreamTrackProcessor

I can't seem to be able to process each frames of a video only once using MediaStreamTrackProcessor.

I'm building a web app where I need to process each frames individually, only once, at a pre-defined pace.

Initially, I was using requestAnimationFrame, calculating the frame displayed using a known constant (FPS) and the currentTime attribute of the video element, and making sure I only process the frame if not already done. This is not ideal as the browser can sometimes skip video frames and we don't have control over that.

Similarly, I started using requestVideoFrameCallback, which is much better because this API is only called when a new frame was displayed on the screen, but it still has the same limitation where the browser can skip some frames.

I then went and tested the Web Codecs API (MediaStreamTrackProcessor) which is meant to do exactly that, processing frame by frame, but it turns out that I can't figure out how to process each frames ONLY ONCE, the reader stream is emitting at a much higher rate than the video playback, and it seems to emit the same frame multiple times if the video is playing back slowly.

See example here: https://codepen.io/Kevin-Dorion/pen/ZEZpybJ

So, question is, how do I make sure I only process each frame once in the reader?

Upvotes: 0

Views: 486

Answers (0)

Related Questions