Webcam Image Processing in C++ OpenCV is Slow

Question

I want to simulate protanopia view (one of the partial colorblind) with C++ OpenCV using webcam to simulate it. Here the code:

#include "opencv2/opencv.hpp"

using namespace cv;

int main(int, char**)
{
    int i = 0;
    Mat im, im2, kernel3;
    Mat rgb2lms = (Mat_<double>(3, 3) << 17.8824, 43.5161, 4.11935, 3.45565, 27.1554,   3.86714, 0.0299566, 0.184309, 1.46709); //filter1
    Mat lms2lmsp = (Mat_<double>(3, 3) << 0, 2.02344, -2.52581, 0, 1, 0, 0, 0, 1); //filter2
    Mat Result, Result2, op1, op2, op3, op4, op5, op6;
    Vec3b zay, zay2;
    kernel3 = rgb2lms.inv(DECOMP_LU);       //filter 3
    cv::Mat mat(3, 1, CV_64FC1);     //create MAT for matrices multiplication 

Mat frame;
VideoCapture cap(0); // open the default camera
if (!cap.isOpened())  // check if we succeeded
    return -1;
namedWindow("edges", 1);
for (;;)
{

    cap.read(frame); // get a new frame from camera
    if (frame.empty()) continue;
    const int nChannels = frame.channels();
    if (i == 0){
        Result.create(frame.size(), frame.type());
    }
  //Result2.create(frame.size(), frame.type());
    cvtColor(frame, im2, CV_BGR2RGB);   //convert to RGB
    for (int i = 0; i < im2.rows; i++)
    {
        for (int j = 0; j < im2.cols; j++)
            {
                for (int k = 0; k < nChannels; k++)
                    {
                        zay(k) = im2.at<Vec3b>(i, j)[k]; //acces pixel value and put into 3x1 vector zay
                    //put the value in to mat so i can multiplied with easy
                    mat.at <double>(0, 0) = zay[0]; 
                    mat.at <double>(1, 0) = zay[1];
                    mat.at <double>(2, 0) = zay[2];
                    op1 = rgb2lms*mat;     //apply filter1
                    op2 = lms2lmsp*op1;    //apply filter2
                    op3 = kernel3*op2;     //apply filter3

                for (int k = 0; k < nChannels; k++)
                    {
                    Result.at<Vec3b>(i, j)[k] = op3.at<double>(k, 0);   //put the result from vector to mat
                    }

            }
    }

    cvtColor(Result, Result2, CV_RGB2BGR);  //convert back to BGR
    imshow("hasil", Result2);
    if (waitKey(30) >= 0) break;
    i++;
}
// the camera will be deinitialized automatically in VideoCapture destructor
return 0;

}

This code is run, but the image video output is very slow (lagging). i'm very new using C++. My Question:

1. Very slow output because i'm using many iterations?
2. I have tried this process using filter2D (convolution) and the result is different. Is the convolution different with matrix multiplication filter?
3. How to make the output smoothly? Should i use pointer in that code?

Thanks

Answer 1

Filter2D is not the appropriate function for your task. It applies a convolution filter on each channel of the image, while what you want is a linear tranformation matrix applied on each rgb pixel.

The function you are looking for is transform.

First, for better performance, merge your three transformations into a single global transformation matrix:

Mat global_kernel = kernel3*lms2lmsp*rgb2lms;

Then, instead of your for loop, use:

transform(im2, Result, global_kernel);

If you still want to save a few milliseconds, you may also remove the cvtColor function calls by applying your transformation directly from bgr color space. Simply switch the columns of your rgb2lms matrix:

Mat bgr2lms = (Mat_<double>(3, 3) << 4.11935, 43.5161, 17.8824,  3.86714, 27.1554, 3.45565,  1.46709, 0.184309, 0.0299566);

Answer 2

I think your problem is the Mat::at<>() calls, which are quite slow. To speed up, you could try using pointer access to the data instead. This is trickier, but much faster. The OpenCV tutorial shows how to do it.