How to extract timestamps from each frame obtained by USB camera?

Question

This scene is common in real time video proccessing. And I need timestamps to synchronize with other devices.

I have tried cv::VideoCapture, but it can not extract the timestamps frome video stream.

So I have two questions here:

1. Does video stream provided by USB camera indeed contains the timestamp information ?
2. If it has. What should I do to extract it ? A C# solution is best, while C++ is OK.

Addition:

Using these two properties doesn't work:

secCounter = (long) cap.get(CAP_PROP_POS_MSEC);
frameNumber = (long) cap.get(CAP_PROP_POS_FRAMES);

It always gives the following result:

VIDEOIO ERROR: V4L2: getting property #1 is not supported

msecCounter = 0

frameNumber = -1

Answer 1

OpenCV's VideoCapture class is a very high level interface to retrieve frames from a camera, so it "hides" a lot of the details that are necessary to connect to the camera, retrieve frames from the camera, and decode those frames in to a useful color space like BGR. This is nice because you don't have to worry about the details of grabbing frames, but the downside is that you don't have direct access to other data you might want, like the frame number or frame timestamp. That doesn't mean it's impossible to get the data you want, though!

Here's a sample frame grabbing loop that will get you what you want, loosely based on the example code from here. This is in C++.

#include "opencv2/opencv.hpp"
using namespace cv;
int main(int, char**)
{
    VideoCapture cap(0); // open the default camera
    if(!cap.isOpened())  // check if we succeeded
        return -1;

    // TODO: change the width, height, and capture FPS to your desired
    // settings.
    cap.set(CAP_PROP_FRAME_WIDTH, 1920);
    cap.set(CAP_PROP_FRAME_HEIGHT, 1080);
    cap.set(CAP_PROP_FPS, 30);

    Mat frame;
    long msecCounter = 0;
    long frameNumber = 0;

    for(;;)
    {            
        // Instead of cap >> frame; we'll do something different.
        //
        // VideoCapture::grab() tells OpenCV to grab a frame from
        // the camera, but to not worry about all the color conversion
        // and processing to convert that frame into BGR.
        //
        // This means there's less processing overhead, so the time
        // stamp will be more accurate because we are fetching it
        // immediately after.
        //
        // grab() should also wait for the next frame to be available
        // based on the capture FPS that is set, so it's okay to loop
        // continuously over it.

        if(cap.grab())
        {
            msecCounter = (long) cap.get(CAP_PROP_POS_MSEC);
            frameNumber = (long) cap.get(CAP_PROP_POS_FRAMES);

            // VideoCapture::retrieve color converts the image and places
            // it in the Mat that you provide.
            if(cap.retrieve(&frame))
            {
                // Pass the frame and parameters to your processing
                // method.
                ProcessFrame(&frame, msecCounter, frameNumber);
            }
        }

        // TODO: Handle your loop termination condition here
    }
    // the camera will be deinitialized automatically in VideoCapture destructor
    return 0;
}

void ProcessFrame(Mat& frame, long msecCounter, long frameNumber)
{
    // TODO: Make a copy of frame if you are going to process it
    // asynchronously or put it in a buffer or queue and then return
    // control from this function. This is because the reference Mat
    // being passed in is "owned" by the processing loop, and on each
    // iteration it will be destructed, so any references to it will be
    // invalid. Hence, if you do any work async, you need to copy frame.
    //
    // If all your processing happens synchronously in this function,
    // you don't need to make a copy first because the loop is waiting
    // for this function to return.

    // TODO: Your processing logic goes here.
}

If you're using C# and Emgu CV it will look a bit different. I haven't tested this code, but it should work or be very close to the solution.

using System;
using Emgu.CV;
using Emgu.CV.CvEnum;
static class Program
{
    [STAThread]
    static void Main()
    {
        VideoCapture cap = new VideoCapture(0);
        if(!cap.IsOpened)
        {
            return;
        }

        cap.SetCaptureProperty(CapProp.FrameWidth, 1920);
        cap.SetCaptureProperty(CapProp.FrameHeight, 1080);
        cap.SetCaptureProperty(CapProp.Fps, 30);

        Mat frame = new Mat();            
        long msecCounter = 0;
        long frameNumber = 0;

        for(;;)
        {
            if(cap.Grab())
            {
                msecCounter = (long) cap.GetCaptureProperty(CapProp.PosMsec);
                frameNumber = (long) cap.GetCaptureProperty(CapProp.PosFrames);

                if(cap.Retrieve(frame))
                {
                    ProcessFrame(frame, msecCounter, frameNumber);
                }
            }

            // TODO: Determine when to quit the processing loop
        }
    }

    private static void ProcessFrame(Mat frame, long msecCounter, long frameNumber)
    {
        // Again, copy frame here if you're going to queue the frame or
        // do any async processing on it.

        // TODO: Your processing code goes here.
    }
}

Emgu's VideoCapture implementation also allows for asynchronous Grab operations to be done for you, and notifications when a grabbed frame is ready to be used with Retrieve. That looks like this:

using System;
using Emgu.CV;
using Emgu.CV.CvEnum;
static class Program
{
    private static Mat s_frame;
    private static VideoCapture s_cap;
    private static object s_retrieveLock = new object();

    [STAThread]
    static void Main()
    {
        s_cap = new VideoCapture(0);
        if(!s_cap.IsOpened)
        {
            return;
        }

        s_frame = new Mat();

        s_cap.SetCaptureProperty(CapProp.FrameWidth, 1920);
        s_cap.SetCaptureProperty(CapProp.FrameHeight, 1080);
        s_cap.SetCaptureProperty(CapProp.Fps, 30);

        s_cap.ImageGrabbed += FrameIsReady;

        s_cap.Start();

        // TODO: Wait here until you're done with the capture process,
        // the same way you'd determine when to exit the for loop in the
        // above example.

        s_cap.Stop();
        s_cap.ImageGrabbed -= FrameIsReady;
    }

    private static void FrameIsReady(object sender, EventArgs e)
    {
        // This function is being called from VideoCapture's thread,
        // so if you rework this code to run with a UI, be very careful
        // about updating Controls here because that needs to be Invoke'd
        // back to the UI thread.

        // I used a lock here to be extra careful and protect against
        // re-entrancy, but this may not be necessary if Emgu's
        // VideoCapture thread blocks for completion of this event
        // handler.
        lock(s_retrieveLock)
        {
            msecCounter = (long) s_cap.GetCaptureProperty(CapProp.PosMsec);
            frameNumber = (long) s_cap.GetCaptureProperty(CapProp.PosFrames);

            if(s_cap.Retrieve(s_frame))
            {
                ProcessFrame(s_frame, msecCounter, frameNumber);
            }
        }
    }

    private static void ProcessFrame(Mat frame, long msecCounter, long frameNumber)
    {
        // Again, copy frame here if you're going to queue the frame or
        // do any async processing on it.

        // TODO: Your processing code goes here.
    }
}