CODEWITHSUNDEEP
javascripthtmlhtml5-videovideo-processingframe-rate
RocketNuts
RocketNuts

Reputation: 11140

HTML5 video: jump to a specific frame rather than a time position? Or step paused video to next or previous frame?

When I have a HTML5 video object, I can set its currentTime property to jump (seek) to a specific time position.

Is there also a way to jump or seek to a specific frame number?

Or related: is there a way for a paused video to step or skip to the next or previous frame?
I mean when it's currently paused or freezes at some arbitrary position, cause the video to jump to the succeeding or preceding frame.

Upvotes: 3

Views: 4700

Answers (2)

vector001
vector001

Reputation: 31

I have 2 solutions for dealing with frames which require pre-scanning the entire video.

Setup:

let video = document.getElementById("my-video") video is an HTMLMediaElement

Our toolkit:

  • video.play() and video.pause()
  • video.currentTime (in seconds) is the position of our video and the best way to change it. (There is also ).
  • video.duration (in seconds) is the total length of the video.
  • video.playbackRate default 1.0, max 16.0 (at least for me).
  • video.getVideoPlaybackQuality() gives counts for totalVideoFrames, droppedVideoFrames, and corruptedVideoFrames
  • let vfcId = video.requestVideoFrameCallback((now,metadata)=>{}) and video.cancelVideoFrameCallback(vfcId) calls the function when a frame is displayed.
  • video.onended = (event) => {} (essentially an event listener for when the video ends.)

Other useless tools:

  • video.fastSeek() which is less accurate but faster than currentTime.
  • video.muted = true might prevent from being annoying
  • video.ontimeupdate / "timeupdate" event. Useless because activates only every few frames, which isn't good enough.

Goal: Create a working goToFrame(n) function.

Solution 1: VideoPlaybackQuality (speed over (a bit of) quality)

Strategy: Find the frame rate and use it to calculate the frame times.

video.getVideoPlaybackQuality().totalVideoFrames gives us the total number of frames so far played in the video, so we can't skip around in the video and have to watch it whole. Still, totalVideoFrames / video.duration will give us the frame rate.

let framerate = None;
function getFrameRate() {
  video.playbackRate = 16;
  video.currentTime = 0;
  video.play();
  video.onended = (event) => { 
    framerate = video.getVideoPlaybackQuality().totalVideoFrames / video.currentTime;
    video.onended = () => (null); //could be cleaner
};
}

function timeForFrame(n) {
  return Math.round(n/framerate *1000)/1000;
}

function goToFrame(n) {
  video.currentTime = timeForFrame(n);
}

Notice that I set the playbackRate to x16 to make the scanning faster. This makes the video player drop frames but, they are still counted by totalVideoFrames.

Advantages: Relatively fast at x16 playback, and you can make the pre-scanning into a loading-preview thing. For a long video, you can estimate the frame rate from the first few seconds using a Timeout.

Disadvantages: If video frame rate varies over time, this method could be inaccurate.

Solution 2: VideoFrameCallback (quality but really slow)

Strategy: create a list (frame_times) with the currentTime for every frame.

VideoFrameCallback calls back on every displayed frame, with lots of time metadata. metadata.mediaTime is the currentTime for the frame.

Note: requestVideoFrameCallback is not an EventListener you need to ask for it again and again for every next frame.

let frame_times = [];

let vfcId = 0;
function getFrametimes() {
  video.playbackRate = 1; // will start dropping frames if higher.
  video.currentTime = 0;
  video.requestVideoFrameCallback(frameTimePusher);
  video.play();
  video.onended = {
    video.cancelVideoFrameCallback(vfcId);
    video.onended = () => (null);
  }
}
function frameTimePusher(now,metadata) {
  frame_times.push(metadata.mediaTime); //mediaTime is currentTime
  video.requestVideoFrameCallback(frameTimePusher); // !! you need to request again.
}
function goToFrame(n) {
  video.currentTime = frame_times[n];
}

Notice that the playbackRate is set to x1. This is because if it is set any higher, the video will start dropping frames and VideoFrameCallback only activates for non-dropped frames.

Advantages: Very accurate (if running at x1 playback rate)

Disadvantages: Slow (if you are running at x1 playback rate)

Solution 1 is arguably better. It also makes determining your current frame easy.

Upvotes: 1

Mick
Mick

Reputation: 25481

As mentioned in the comments, and AFAIK, there is no standard defined to support frame accurate seeking for the HTML5 video tag itself, other than the mentioned 'seekToNextFrame' which is not widely supported as you can see here: https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/seekToNextFrame

There are commercial Javascript HTMl5 players which provide this functionality - e.g.:

You'll also find discussion on this topic in the open source Shaka player discussion groups.

The reason it is not straightforward is that it usually requires the player or the browser (or something client side) to first find the nearest 'I' frame and then to decode forwards and/or backwards from that to get to the exact frame you want. It's certainly technically possible, as proven by the above example implementations, but I would guess there has not been the demand in the browser or open source player community to prioritise this functionality.

Upvotes: 2

Related Questions