What is the best feature detection?

Question

We have two images, one as a reference and another image which we want to align like the reference using Matlab. In order to do this, we need to find similar points in both images and than calculate the kernel matrix using the Least Square method for the transformation like this code:

clear all; close all; clc; imtool close all;
      I1 = rgb2gray(im);
      I2 = rgb2gray(ref);
      points1 = detectSURFFeatures(I1);
      points2 = detectSURFFeatures(I2);
      [features1,valid_points1] = extractFeatures(I1,points1,'SURFSize',64);
      [features2,valid_points2] = extractFeatures(I2,points2,'SURFSize',64);
      indexPairs = matchFeatures(features1,features2);
      matchedPoints1 = valid_points1(indexPairs(:,1),:);
      matchedPoints2 = valid_points2(indexPairs(:,2),:);
      [tform, correct1,correct2] = estimateGeometricTransform(matchedPoints1,matchedPoints2,'projective','MaxNumTrials',100000,'Confidence',99.9999,'MaxDistance',4);
      sourcepoints = correct1.Location;
      targetpoints = correct2.Location;
      sizenum = size(sourcepoints,1);
      x_source = sourcepoints(:,1);
      y_source = sourcepoints(:,2);
      x_target = targetpoints(:,1);
      y_target = targetpoints(:,2);
      zero_vec = zeros([sizenum,1]);
      one_vec = ones([sizenum,1]);
      H = [x_source,y_source,one_vec,zero_vec,zero_vec,zero_vec,-x_source.*x_target,-y_source.*x_target;...
           zero_vec,zero_vec,zero_vec,x_source,y_source,one_vec,-x_source.*y_target,-y_source.*y_target];
      Y = [x_target;y_target];
      variables = (inv(H'*H))*H'*Y;
      variables(9) = 1;
      kernel = reshape(variables,3,3)';

In this code we don't care much about the speed, the most important thing to us is the accuracy.

1) In here we are using the Surf method and we try to understand if this is the best method for this task, or maybe we should use other feature detection like HOG or FAST etc?

2) We try to understand what are the differences between each feature detector and when to use each one of them?

Thanks in advance.

Answer 1

There is no such thing as "best" feature detector, absolutely speaking, the quality of the matching strongly depends on the specific image and moreover on the parameters you use to configure your detector, altrough there are feature detectors that objectively have a wider range of conditions in which they work well

• SURF and SIFT are often considered to be the best feature detectors out there, for good reasons, they are very robust and very fast in most situations, the only scenario I found so far in which they show their weaknesses is with highly detailed targets (electrical boards for instance), but keep in mind that SURF and SIFT are both patent protected so you will have to pay lots of money if your goal involves commercial use

• AKAZE (https://github.com/pablofdezalc/akaze) is a valid free to use alternative to SURF/SIFT: its robustness is similar to SIFT/SURF in most cases, as well as its performances, the only constraint here is that you need a 64bit architecture to run AKAZE with good performances, I found out that on 32bit architectures the performaces drop drastically

• FAST is, as its name suggests, very fast, and very "greedy", it extracts a lot of keypoints compared to other detectors, but it is not rotation invariant (meaning that it won't work if the target is rotated respect the reference image). Also it is a just a detector, so you will have to describe the keypoints you extract with some other descriptor, and the matching robustness will depend on this, my advice is to try FREAK or ORB as descriptor both of which gave me good results with FAST

• BRIEF has good performance and does extract a lower number of keypoints than FAST, just like FAST it is not rotation invariant and does not have a corresponding descriptor

• ORB is basically an evolution of the previous 2 detectors (ORB stands for Oriented fast and Rotated Brief) that is rotation invariant and also implements its own descriptor, this is probably the best choice for general purposes, it is free to use and its robustness is comparable to SIFT/SURF/AKAZE while the performances slightly overcomes them (using default parameters), altrough the robustnes is actually a little inferior in most of the situations, there are specific scenarios in which it overcomes SURF/SIFT/AKAZE (once again electrical boards for instance)

• BRISK has a behavior very similar to ORB with a little more cpu load, since ORB in most cases works better in both terms of robustness and performances people usually end up using ORB instead

• HOG (which you mentioned) is actually not meant to be used with features matching, it is more likely suitable for deep learning classification

those are the most popular detectors (and descriptors), there are lots of other detectors (AGAST, GFTT, MSER, STAR, etc.) and descriptors (LATCH, LUCID, DAISY, etc.) which are, in most conditions, inferior to the ones I listed but that can probabily fit specific situations better

If you are intrested in testing those detectors/descriptors robustness and performances in a quick way I suggest you to download and install Find-Object (http://introlab.github.io/find-object/) which lets you benchmark those algorithms through target image matching providing a confortable interface to do that with ease, also the source code is available on github if you are intrested in it (https://github.com/introlab/find-object)

Answer 2

Short summary on your points:

1) SIFT and SURF are based on the Histogram of Gradients (HoG) that is each gradient in the the pixel patch is calculated and these calculations cost time. SIFT and SURF are patent protected. Eventhough it takes time it gives you the best results when time, power consumption and computational costs on the device architecture doesn't matter for you.

2) It depends on where you are going to use the application. It it a system based application or for the hand held devices (mobiles) or for the smart glasses?

SIFT and SURF yield poor results on mobile phones and on the smart glasses. FAST is not a rotation invariant. BRISK needs high computational GPU. If there is no scenario where you are going to use this it's quite hard to suggest which would be the best.

You can find more detailed answers from the below post

Stack Overflow