OpenCV - Basic Operations - Performance Issue [in Mode: Release]

Question

I might discovered a huge performance issue with OpenCV's own implementation of matrix multiplication / summation, and wanted to check with you guys if I maybe missing something:

In advance: All runs were done in (OpenCV's) Release Mode.

Setup:

(a) I'll do 10 million times a matrix-vector multiplication with a 3-by-3 matrix and a 3-by-1 vector. The implementation follows the code: res = mat * vec;

(b) I'll do the same with my own implementation of accessing the elements individually and then doing the multiplication process using pointer-arithmetic. [basically just multiplying out the process and writing down the equations for each row for the result vector]

I tested these variants with the compiler flags -O0, -O1, -O2, -O3, -Ofast and for OpenCV 3.1 & 3.2.

The timings are done using chrono (high_resolution_clock) on Ubuntu 16.04.

Findings:

In all cases the non-optimized method (b) outperforms the OpenCV method (a) by a factor of ~100 to ~1000.

Question:

How can that be the case? Shouldn't OpenCV be optimized for these kinds of procedures? Should I raise an issue on Github, or is there something I'm totally missing?

Code: [Ready to copy and test on your machine]

#include <chrono>
#include <iostream>

#include "opencv2/core/cvstd.hpp"
#include "opencv2/core.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"



int main()
{

    // 1. Setup:

    std::vector<std::chrono::high_resolution_clock::time_point> timestamp_vec_start(2);
    std::vector<std::chrono::high_resolution_clock::time_point> timestamp_vec_end(2);
    std::vector<double> timestamp_vec_total(2);


    cv::Mat test_mat = (cv::Mat_<float>(3,3) <<  0.023, 232.33, 0.545, 
                                                 22.22, 0.1123, 4.444,
                                                 0.012, 3.4521, 0.202);

    cv::Mat test_vec = (cv::Mat_<float>(3,1) <<  5.77, 
                                                 1.20,
                                                 0.03);

    cv::Mat result_1 = cv::Mat(3, 1, CV_32FC1);
    cv::Mat result_2 = cv::Mat(3, 1, CV_32FC1);

    cv::Mat temp_test_mat_results = cv::Mat(3, 3, CV_32FC1);
    cv::Mat temp_test_vec_results = cv::Mat(3, 1, CV_32FC1);

    auto ptr_test_mat_res_0 = temp_test_mat_results.ptr<float>(0);
    auto ptr_test_mat_res_1 = temp_test_mat_results.ptr<float>(1);
    auto ptr_test_mat_res_2 = temp_test_mat_results.ptr<float>(2);

    auto ptr_test_vec_res_0 = temp_test_vec_results.ptr<float>(0);
    auto ptr_test_vec_res_1 = temp_test_vec_results.ptr<float>(1);
    auto ptr_test_vec_res_2 = temp_test_vec_results.ptr<float>(2);

    auto ptr_res_0 = result_2.ptr<float>(0);
    auto ptr_res_1 = result_2.ptr<float>(1);
    auto ptr_res_2 = result_2.ptr<float>(2);





    // 2. OpenCV Basic Matrix Operations:

    timestamp_vec_start[0] = std::chrono::high_resolution_clock::now();

    for(int i = 0; i < 10000000; ++i)
    {
        // factor of up to 5000 here:
        // result_1 = (test_mat + test_mat + test_mat) * (test_vec + test_vec);

        // factor of 30~100 here:
        result_1 = test_mat * test_vec;
    }

    timestamp_vec_end[0]   = std::chrono::high_resolution_clock::now();
    timestamp_vec_total[0] = static_cast<double>(std::chrono::duration_cast<std::chrono::microseconds>(timestamp_vec_end[0] - timestamp_vec_start[0]).count());





    // 3. Pixel-Wise Operations:

    timestamp_vec_start[1] = std::chrono::high_resolution_clock::now();

    for(int i = 0; i < 10000000; ++i)
    {
        auto ptr_test_mat_0 = test_mat.ptr<float>(0);
        auto ptr_test_mat_1 = test_mat.ptr<float>(1);
        auto ptr_test_mat_2 = test_mat.ptr<float>(2);

        auto ptr_test_vec_0 = test_vec.ptr<float>(0);
        auto ptr_test_vec_1 = test_vec.ptr<float>(1);
        auto ptr_test_vec_2 = test_vec.ptr<float>(2);


        ptr_test_mat_res_0[0] = ptr_test_mat_0[0] + ptr_test_mat_0[0] + ptr_test_mat_0[0];
        ptr_test_mat_res_0[1] = ptr_test_mat_0[1] + ptr_test_mat_0[1] + ptr_test_mat_0[1];
        ptr_test_mat_res_0[2] = ptr_test_mat_0[2] + ptr_test_mat_0[2] + ptr_test_mat_0[2];

        ptr_test_mat_res_1[0] = ptr_test_mat_1[0] + ptr_test_mat_1[0] + ptr_test_mat_1[0];
        ptr_test_mat_res_1[1] = ptr_test_mat_1[1] + ptr_test_mat_1[1] + ptr_test_mat_1[1];
        ptr_test_mat_res_1[2] = ptr_test_mat_1[2] + ptr_test_mat_1[2] + ptr_test_mat_1[2];

        ptr_test_mat_res_2[0] = ptr_test_mat_2[0] + ptr_test_mat_2[0] + ptr_test_mat_2[0];
        ptr_test_mat_res_2[1] = ptr_test_mat_2[1] + ptr_test_mat_2[1] + ptr_test_mat_2[1];
        ptr_test_mat_res_2[2] = ptr_test_mat_2[2] + ptr_test_mat_2[2] + ptr_test_mat_2[2];

        ptr_test_vec_res_0[0] = ptr_test_vec_0[0] + ptr_test_vec_0[0];
        ptr_test_vec_res_1[0] = ptr_test_vec_1[0] + ptr_test_vec_1[0];
        ptr_test_vec_res_2[0] = ptr_test_vec_2[0] + ptr_test_vec_2[0];

        ptr_res_0[0] = ptr_test_mat_res_0[0]*ptr_test_vec_res_0[0] + ptr_test_mat_res_0[1]*ptr_test_vec_res_1[0] + ptr_test_mat_res_0[2]*ptr_test_vec_res_2[0];
        ptr_res_1[0] = ptr_test_mat_res_1[0]*ptr_test_vec_res_0[0] + ptr_test_mat_res_1[1]*ptr_test_vec_res_1[0] + ptr_test_mat_res_1[2]*ptr_test_vec_res_2[0];
        ptr_res_2[0] = ptr_test_mat_res_2[0]*ptr_test_vec_res_0[0] + ptr_test_mat_res_2[1]*ptr_test_vec_res_1[0] + ptr_test_mat_res_2[2]*ptr_test_vec_res_2[0];
    }

    timestamp_vec_end[1]   = std::chrono::high_resolution_clock::now();
    timestamp_vec_total[1] = static_cast<double>(std::chrono::duration_cast<std::chrono::microseconds>(timestamp_vec_end[1] - timestamp_vec_start[1]).count());





    // 4. Printout Timing Results:

    std::cout << "\n\nTimings:\n\n";
    std::cout << "Time spent in OpenCV's implementation:      "  << timestamp_vec_total[0]/1000.0 << " ms.\n";
    std::cout << "Time spent in element-wise implementation:  "  << timestamp_vec_total[1]/1000.0 << " ms.\n\n";

    std::cin.get();

    return 0;
}

Answer 1

OpenCV is not optimized for small matrix operations.
You can reduce your overhead a little by not allocating a new Matrix for the result inside the loop by using cv::gemm

But if small matrix operations are a bottleneck for you I recommend using Eigen.

Using a quick Eigen implementation like:

Eigen::Matrix3d mat;
mat << 0.023, 232.33, 0.545,
    22.22, 0.1123, 4.444,
    0.012, 3.4521, 0.202;

Eigen::Vector3d vec3;
vec3 << 5.77,
    1.20,
    0.03;

Eigen::Vector3d result_e;

for (int i = 0; i < 10000000; ++i)
{
    result_e = (mat *3 ) * (vec3 *2);
}

gives me the following numbers with VS2015 (obviously the difference might be less dramatic in GCC or Clang):

Timings:

Time spent in OpenCV's implementation:      2384.45 ms.
Time spent in element-wise implementation:  78.653 ms.
Time spent in Eigen implementation:         36.088 ms.