Pattern Matching - Find reference object in second image [OpenCV?]

Question

I have a reference b/w image that contains a reference object (a coin for example). The object is marked by the user, that is the region of interest.

Now I want to analyze other images and find the position and rotation of that object or similar objects. The object is arbitrarily placed under the camera, but is never scaled and the viewing angle is always 90 degrees.

I have evaluated a commercial library that does exactly what I want: Euresys EasyFind

Below you can find example images of the task at hand. The current implementation uses Feature Detection of OpenCV and is not working flawlessly.

Template:

A match is found for the very same coin:

The match fails for slightly different coins:

The feature detection seems to be the wrong approach. I need to simply the object somehow. But if I do that (Blur, Canny, CornerHarris) feature detection does not work at all.

Any advice for a solid approach is much appreciated. An alternative libary suggestion would be great as well.

Answer 1

Since you have tried quite a number of possible techniques, I would request you to go through the following links (may be you might have gone through!!!)

1. comparision of all feature detectors and descriptors
2. combination of surf,FREAK and brisk

your 3rd image which fails is having low contrast and it is little tricky to match perfectly with the rest two...So I did a contrast adjustment and I get the following match with Orb Feature detector and Orb Descriptor Extractor..I applied contrast adjustment to all the images before feature detection.

IMAGE 1 WITH IMAGE 3

IMAGE 2 WITH IMAGE 3

IMAGE 1 WITH IMAGE 2 (THIS COMBINATION WORKS GOOD WITH ALL DETECTOR/EXTRACTOR PAIRS)

For matching I have used BruteForceMatcher<Hamming> matcher Although the points are localised orientation can be quite nicely guessed. One needs to use more then one technique and do some circle detection first to limit the feature detection to as minimum ROI as possible. Plus orientation of the detected points with respect to the centre of the circle gives the new orientation info easily. With reference to the 1st link and 2nd link you can notice SURF and BRIEF are quite resistant to change in intensity of light and blurring. So a combination of SURF and BRIEF is what you can also try.

Answer 2

In general you have the following steps were changes could affect the system performance:

1. Detector and its configuration
2. Descriptor and its configuration
3. Matcher and its configuration
4. RANSAC parameters (or whatever you're using)

You could try MSER (maximally stable extremal region) as an detector. This detector is known not to find as much keypoints as other detectors like SIFT/SURF. But they are very stable and unique. But I would try a different descriptor for matching. Perhaps FREAK (Fast Retina Keypoint).

The Filtering with the RANSAC-algorithm should do but you can always tweak by setting a distance you like. If RANSAC doesn't work well then try switching the default threshold until it seems to work better.

Another thing you can change is the matcher. FLANN is very good but only approximates the results and receives great performs when searching in high-dimensional spaces. But if performance doesn't concern you then try a BruteForceMatcher with euclidean or Mahalanobis distance.

[EDIT-1]: *Now I'm trying to give you additional could-help-your-suggestions =) * Kampel et al. describe in their article "Image-Based Retrieval and Identification of Ancient Coins" some additional approaches to give an acient coin a shape-descriptor to enhance the describing features of items.

Another problem could be RANSAC which is a very restrictive approach to eliminate outliers. The method Mat findHomography(InputArray srcPoints, InputArray dstPoints, int method=0, double ransacReprojThreshold=3, OutputArray mask=noArray() ) has additional parameters (click this link). You could use all points, RANSAC or a Least-Median robust method.

Or you try to make a robust calculation what distance might be a good threshold to retrieve good matches. The standard tutorial uses two times the minimal found distance which is a start but not very robust cause it depends on the distance of the current matches.

There are several other descriptors/detectors which are not included in the standard OpenCV. Lindbjerg describes in his paper "Finding the Best Feature Detector-Descriptor Combination" that MSER works finde with a DAISY-Descriptor. Or you could try AGAST or BRISK. One real big drawback is that they are all not included in the standard OpenCV distribution. There are some implementations but you have to integrate them which could mean a lot of work.

Answer 3

Applying a key point detector in the inputs without performing any kind of pre-processing is a very poor way to find relevant interesting matching points.

Given your images, Mathematical Morphology provides good tools to pre-process them and hopefully obtain better matches at a later step. One of these tools is called morphological reconstruction, and its generalization called Levelings, which will merge and extend flat zones based on a marker image. As you can see, your image is very noise in the sense that there are valleys (those dark points) everywhere, so you want to suppress most of that. Implementing Levelings is not hard at all, but since I'm not aware of freely available tools (actually there is a large framework developed mostly in France that I believe contains it, but I don't remember the name), let us stick with the more standard morphological reconstruction. We also don't have a marker image, but that is easy to construct: erode your image with some structuring element and apply morphological reconstruction, that is called geodesic opening by some authors. This will suppress some of your noisy valleys.

At this point your image might be good to use with some keypoint detector, but I would suggest binarizing it since what you want to match is irrelevant to the grayscale tones. The typical automatic method for this is given by Otsu, but there are others like Kapur's method which goes for histogram entropy minimization. I didn't use Otsu simply because it is very common, for some Stackoverflow-novelty I went for this other method :P (yes, very bad reason). Lastly, after binarization, you can continue processing your image. A single morphological closing might be enough here, as you want to remove some remaining noise points (which are not necessarily disconnected from the components, so removing components is potentially a bad choice now).

To keep it short, here is geodesic opening and the final pre-processing (using the points above) for your first image:

And here is the final result for your other two images (applying exactly the same process, no constant modification or anything else):

If I now proceed and match these using the typical SURF method, I get mostly perfect matches making the other problem (orientating one in relation to the other) very easy to solve. I'm not including the results for the matchings I get (unless someone want to compare the results) because I believe you should get the same result using OpenCV or any other correct implementation now.

If it matters, here is the single line in Mathematica to achieve the pre-processing discussed above:

g = Closing[Binarize[GeodesicOpening[f, DiskMatrix[5]], Method -> "Entropy"], 1]

where f is the input image in grayscale, and g is the resulting binary image.

Answer 4

I believe that what would work best for you is some version of brute-force search, similar to cvMatchTemplate. All feature-extraction-based searches are quite noise-sensitive; unless you have some a-priori reason to know that certain types of features are going to be prominent in all of the images of interest (for example: outline circles/ovals for coins), then feature extraction is unlikely to give you good performance. The problem with brute-force matching is that it is scale and rotation sensitive. Perhaps you can look at the literature for scale-free matching, and/or brute-force possible scales and rotations as well. For the example image you gave, it would actually match itself pretty well if rotated by a few degrees and scaled by a few percent; you "only" need a few hundred calls to cvMatchTemplate to find a rotated, scaled version.

Answer 5

The question is quite broad and there are many ways depending on what exactly you want to achieve. You can use the Hough Transform to detect the coin because it's round, though it might detect many round elements (so it depends on whether you have other similar things in view).

More generally, you need to use feature detector (SURF, ORB, FAST, Shi-Tomassi), do extraction and matching between a reference frame that has the coin and the frame in which you're looking for it. Then, you can use the Homography Transform to ensure that the points you've found are the same, and hence you've found your coin.

Take a look at this example:

Features2D + Homography to find a known object