Getting timestamp of each frame in a video

Question

I have recorded several videos from the front cam of my tablet with an Android 5.2 application I have written. I have stored the start timestamp in milliseconds (Unix time) for each video.

Unfortunately each video has a different framerate (ranging from 20 to 30). With OpenCV I'm able to get the framerate for each video:

import cv2
video = cv2.VideoCapture(videoFile)
fps = video.get(cv2.CAP_PROP_FPS)

This works well and theoretically I could just add 1000/fps (due to milliseconds) for each frame in the video. But this assumes that the framerate remains stable throughout the whole recording. I don't know if this is the case.

Is there a possibility in Python to get the timestamp (in milliseconds) of each frame in the video independent of the framerate?

Answer 1

I have use moviepy to get time in seconds of individual frame

pip install moviepy

import sys
import numpy as np
import cv2
import moviepy.editor as mpy
from matplotlib import pyplot as plt

vid = mpy.VideoFileClip('input_video\\v3.mp4')

for i, (tstamp, frame) in enumerate(vid.iter_frames(with_times=True)):
    print(tstamp%60)
    plt.imshow(frame)
    plt.show()

Answer 2

I did some test with multiple library.

• decord
• ffprobe
• MoviePy
• PyAV
• pyffms2
• OpenCV

import av
import cv2
import json
import os
import shutil
import sys
import subprocess
import time
from decimal import Decimal
from decord import VideoReader
from ffms2 import VideoSource
from moviepy.editor import VideoFileClip
from typing import List


def with_movie_py(video: str) -> List[int]:
    """
    Link: https://pypi.org/project/moviepy/
    My comments:
        The timestamps I get are not good compared to gMKVExtractGUI or ffms2. (I only tried with VFR video)

    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    vid = VideoFileClip(video)

    timestamps = [
        round(tstamp * 1000) for tstamp, frame in vid.iter_frames(with_times=True)
    ]

    return timestamps


def with_cv2(video: str) -> List[int]:
    """
    Link: https://pypi.org/project/opencv-python/
    My comments:
        I don't know why, but the last 4 or 5 timestamps are equal to 0 when they should not.
        Also, cv2 is slow. It took my computer 132 seconds to process the video.


    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    timestamps = []
    cap = cv2.VideoCapture(video)

    while cap.isOpened():
        frame_exists, curr_frame = cap.read()
        if frame_exists:
            timestamps.append(round(cap.get(cv2.CAP_PROP_POS_MSEC)))
        else:
            break

    cap.release()

    return timestamps


def with_pyffms2(video: str) -> List[int]:
    """
    Link: https://pypi.org/project/ffms2/
    My comments:
        Works really well, but it doesn't install ffms2 automatically, so you need to do it by yourself.
        The easiest way is to install Vapoursynth and use it to install ffms2.
        Also, the library doesn't seems to be really maintained.

    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    video_source = VideoSource(video)

    # You can also do: video_source.track.timecodes
    timestamps = [
        int(
            (frame.PTS * video_source.track.time_base.numerator)
            / video_source.track.time_base.denominator
        )
        for frame in video_source.track.frame_info_list
    ]

    return timestamps


def with_decord(video: str) -> List[int]:
    """
    Link: https://github.com/dmlc/decord
    My comments:
        Works really well, but it seems to only work with mkv and mp4 file.
        Important, Decord seems to automatically normalise the timestamps which can cause many issue: https://github.com/dmlc/decord/issues/238
        Mp4 file can have a +- 1 ms difference with ffms2, but it is acceptable.

    Parameters:
        video (str): Video path
    Returns:
        List of timestamps in ms
    """
    vr = VideoReader(video)

    timestamps = vr.get_frame_timestamp(range(len(vr)))
    timestamps = (timestamps[:, 0] * 1000).round().astype(int).tolist()

    return timestamps


def with_pyav(video: str, index: int = 0) -> List[int]:
    """
    Link: https://pypi.org/project/av/
    My comments:
        Works really well, but it is slower than ffprobe.
        The big advantage is that ffmpeg does not have to be installed on the computer, because pyav installs it automatically

    Parameters:
        video (str): Video path
        index (int): Stream index of the video.
    Returns:
        List of timestamps in ms
    """
    container = av.open(video)
    video = container.streams.get(index)[0]

    if video.type != "video":
            raise ValueError(
                f'The index {index} is not a video stream. It is an {video.type} stream.'
            )

    av_timestamps = [
        int(packet.pts * video.time_base * 1000) for packet in container.demux(video) if packet.pts is not None
    ]

    container.close()
    av_timestamps.sort()

    return av_timestamps


def with_ffprobe(video_path: str, index: int = 0) -> List[int]:
    """
    Link: https://ffmpeg.org/ffprobe.html
    My comments:
        Works really well, but the user need to have FFMpeg in his environment variables.

    Parameters:
        video (str): Video path
        index (int): Index of the stream of the video
    Returns:
        List of timestamps in ms
    """

    def get_pts(packets) -> List[int]:
        pts: List[int] = []

        for packet in packets:
            pts.append(int(Decimal(packet["pts_time"]) * 1000))

        pts.sort()
        return pts

    # Verify if ffprobe is installed
    if shutil.which("ffprobe") is None:
        raise Exception("ffprobe is not in the environment variable.")

    # Getting video absolute path and checking for its existance
    if not os.path.isabs(video_path):
        dirname = os.path.dirname(os.path.abspath(sys.argv[0]))
        video_path = os.path.join(dirname, video_path)
    if not os.path.isfile(video_path):
        raise FileNotFoundError(f'Invalid path for the video file: "{video_path}"')

    cmd = f'ffprobe -select_streams {index} -show_entries packet=pts_time:stream=codec_type "{video_path}" -print_format json'
    ffprobeOutput = subprocess.run(cmd, capture_output=True, text=True)
    ffprobeOutput = json.loads(ffprobeOutput.stdout)

    if len(ffprobeOutput) == 0:
        raise Exception(
            f"The file {video_path} is not a video file or the file does not exist."
        )

    if len(ffprobeOutput["streams"]) == 0:
        raise ValueError(f"The index {index} is not in the file {video_path}.")

    if ffprobeOutput["streams"][0]["codec_type"] != "video":
        raise ValueError(
            f'The index {index} is not a video stream. It is an {ffprobeOutput["streams"][0]["codec_type"]} stream.'
        )

    return get_pts(ffprobeOutput["packets"])


def main():
    video = r"WRITE_YOUR_VIDEO_PATH"

    start = time.process_time()
    movie_py_timestamps = with_movie_py(video)
    print(f"With Movie py {time.process_time() - start} seconds")

    start = time.process_time()
    cv2_timestamps = with_cv2(video)
    print(f"With cv2 {time.process_time() - start} seconds")

    start = time.process_time()
    ffms2_timestamps = with_pyffms2(video)
    print(f"With ffms2 {time.process_time() - start} seconds")

    start = time.process_time()
    decord_timestamps = with_decord(video)
    print(f"With decord {time.process_time() - start} seconds")

    start = time.process_time()
    av_timestamps = with_pyav(video)
    print(f"With av {time.process_time() - start} seconds")

    start = time.process_time()
    ffprobe_timestamps = with_ffprobe(video)
    print(f"With ffprobe {time.process_time() - start} seconds")


if __name__ == "__main__":
    main()

Here is how much times it took to get the timestamps for an mp4 of 24 minutes.

With Movie py 11.421875 seconds
With cv2 131.890625 seconds
With ffms2 0.640625 seconds
With decord 0.328125 seconds
With av 0.6875 seconds
With ffprobe 0.21875 seconds

Answer 3

This is a simplified version that just reads in a video and prints out a frame number with its timestamp.

import cv2

cap = cv2.VideoCapture('path_to_video/video_filename.avi')

frame_no = 0
while(cap.isOpened()):
    frame_exists, curr_frame = cap.read()
    if frame_exists:
        print("for frame : " + str(frame_no) + "   timestamp is: ", str(cap.get(cv2.CAP_PROP_POS_MSEC)))
    else:
        break
    frame_no += 1

cap.release()

This gives an output that looks like this:

for frame : 0   timestamp is:  0.0
for frame : 1   timestamp is:  40.0
for frame : 2   timestamp is:  80.0
for frame : 3   timestamp is:  120.0
for frame : 4   timestamp is:  160.0
for frame : 5   timestamp is:  200.0
for frame : 6   timestamp is:  240.0
for frame : 7   timestamp is:  280.0
for frame : 8   timestamp is:  320.0
for frame : 9   timestamp is:  360.0
for frame : 10   timestamp is:  400.0
for frame : 11   timestamp is:  440.0
for frame : 12   timestamp is:  480.0
...

Answer 4

Normally these camera's have a rolling shutter, this implies that the image is scanned line by line, so strickly speaking one cannot put one time stamp on the image. I've been working on the synchronisation of multiple rolling shutter camera's (iPhone 6) using an exactly timed (ns-scale) led-flash. I found that the framerate is variable (nominal 240 fps at high speed, but varies between 239,something and 241,something. Mutual synching can be done up to 1/500000 s, but this requires a special setup. If you are interested I can send you some documentation (I'm afraid my software is in Matlab, so no readily available python code)

Answer 5

You want cv2.CAP_PROP_POS_MSEC. See all the different capture properties here.

Edit: Actually, as Dan Mašek pointed out to me, when you grab that property, it looks like OpenCV is exactly doing that calculation (at least assuming you're using FFMPEG):

case CV_FFMPEG_CAP_PROP_POS_MSEC:
    return 1000.0*(double)frame_number/get_fps();

So it seems you're always going to rely on a constant frame rate assumption. However, even assuming a constant frame rate, it's important that you multiply by the frame number and not just keep adding 1000/fps. Errors will build up when you're repeatedly adding floats which, over a long video, can make a big difference. For example:

import cv2

cap = cv2.VideoCapture('vancouver2.mp4')
fps = cap.get(cv2.CAP_PROP_FPS)

timestamps = [cap.get(cv2.CAP_PROP_POS_MSEC)]
calc_timestamps = [0.0]

while(cap.isOpened()):
    frame_exists, curr_frame = cap.read()
    if frame_exists:
        timestamps.append(cap.get(cv2.CAP_PROP_POS_MSEC))
        calc_timestamps.append(calc_timestamps[-1] + 1000/fps)
    else:
        break

cap.release()

for i, (ts, cts) in enumerate(zip(timestamps, calc_timestamps)):
    print('Frame %d difference:'%i, abs(ts - cts))

Frame 0 difference: 0.0
Frame 1 difference: 0.0
Frame 2 difference: 0.0
Frame 3 difference: 1.4210854715202004e-14
Frame 4 difference: 0.011111111111091532
Frame 5 difference: 0.011111111111091532
Frame 6 difference: 0.011111111111091532
Frame 7 difference: 0.011111111111119953
Frame 8 difference: 0.022222222222183063
Frame 9 difference: 0.022222222222183063
...
Frame 294 difference: 0.8111111111411446

This is of course in milliseconds, so maybe it doesn't seem that big. But here I'm almost 1ms off in the calculation, and this is just for an 11-second video. And anyways, using this property is just easier.