Efficiently calculating distance of pixels to their corresponding epipolar lines

Question

I'm trying to calculate the distance of two corresponding pixels in a stereo camera setup to their respective epipolar lines. The code I Implemented for this purpose looks is the following:

#include <iostream>
#include <vector>
#include <opencv2/calib3d/calib3d.hpp>

float calculateDistanceToEpiLineSum(const cv::Mat2f& left_candidate, const cv::Mat2f& right_candidate, const cv::Matx33f& fundamental_mat) {

  // Calculate epipolar lines
  cv::Mat epiLineRight=cv::Mat(1,3,CV_32FC1);
  cv::Mat epiLineLeft=cv::Mat(1,3,CV_32FC1);
  cv::computeCorrespondEpilines(left_candidate,2,fundamental_mat,epiLineRight);
  cv::computeCorrespondEpilines(right_candidate,1,fundamental_mat,epiLineLeft);

  // Calculate distances of the image points to their corresponding epipolar line
  float distance_left_im=std::abs(epiLineLeft.at<float>(0)*left_candidate[0][0][0]+
                                   epiLineLeft.at<float>(1)*left_candidate[0][0][1]+
                                   epiLineLeft.at<float>(2))/
      std::sqrt(std::pow(epiLineLeft.at<float>(0),2.f)+std::pow(epiLineLeft.at<float>(1),2.f));

  float distance_right_im=std::abs(epiLineRight.at<float>(0)*right_candidate[0][0][0]+
                                   epiLineRight.at<float>(1)*right_candidate[0][0][1]+
                                   epiLineRight.at<float>(2))/
      std::sqrt(std::pow(epiLineRight.at<float>(0),2.f)+std::pow(epiLineRight.at<float>(1),2.f));

  return distance_left_im+distance_right_im;
}

int main()
{
  cv::Matx33f fundamental_mat=cv::Matx33f{-0.000000234008931f,-0.000013193232976f, 0.010025275471910f,
                                          -0.000017896532640f, 0.000009948056751f, 0.414125924093639f,
                                           0.006296743991557f,-0.411007947095269f,-4.695511356888332f};

  cv::Vec2f left_candidate_vec(135.,289.);
  cv::Vec2f right_candidate_vec(205.,311.);
  cv::Mat2f left_candidate(left_candidate_vec);
  cv::Mat2f right_candidate(right_candidate_vec);

  float distance_sum=calculateDistanceToEpiLineSum(left_candidate,right_candidate,fundamental_mat);

  std::cout<<"The sum of the distances equals "<<distance_sum<<" pixels\n";

  return 0;
}

The problem that I'm facing is that I will have to perform this operation possibly thousands of times each second. I am aware that cv::computeCorrespondEpiliness first input can be a vector of pixels which allows a more vectorized approach and would probably speed things up a little. The problem is, that I can not make use of this, because I'm not working with conventional cameras, but event-based sensors and therefore I will receive the pixels asynchronously (instead of receiving frames).
Now I would like to know the following:

1. Are there any major flaws in calculateDistanceToEpiLineSum that influence the performance of the function in a bad way? Is it maybe a good idea, to not use OpenCV functions here, but just implement computeCorrespondEpilines myself?
2. I thought about calculating the epipolar lines "offline" before to save processing time. The problem is, I don't know how I would calculate and safe the epipolar lines efficiently. I thought about calculating the epipolar lines for each pixel of each image and storing the line parameters in 3-tuples according to ax+by+c=0, but assuming a resolution of 480x360 of each camera, that would give me two 480x360x3 matrices which is quite big. Is it viable to do it like this anyway, or is there any better approach?

Answer 1

Questions of time performance are impossible to answer without knowing what your computational constraints are, so what follows is a very rough estimate.

At bottom, computing a distance of one point of a candidate matched pair of pixels from its associated epipolar line cost roughly:

• one matrix multiply = 9 mult, 6 adds = 15 flops
• Normalization = 2 mult, 1 add, 1 square root =~ 10 flops
• Dot product with the line coefficients: 3 mult, 2 add = 5 flops
• Division =~ 4 flops

So about 70 flops for the pair.

What this all means in seconds depends, at a minimum, on your processor clock cycle. A Skylake IA64 can do 16 DP vectorized flops/cycle, so call it 5 cycles for the pair. At 3 GHz that takes less than 2 ns. Taking a generous margin on that, let's call it 10 ns total.

You say you will have to perform this computation "thousand of times" per second. At 10ns per pair, you can do 100 million of them per second.

Given the above, are you sure this particular operation is going to be the bottleneck and not, for example, the I/O from the camera instead (including image decoding)?

Word of advice: learn to use a good microbenchmarking framework to get actual performance numbers for your hardware. I recommend good old Google Benchmark.