OpenCV: Fitting an object into a scene using homography and perspective transform in Java

Question

I'm implementing using Java the OpenCV tutorial for finding an object in a scene using homography http://docs.opencv.org/doc/tutorials/features2d/feature_homography/feature_homography.html#feature-homography

Below is my implementation, where img1 is the scene and img2 is the object

    FeatureDetector detector = FeatureDetector.create(FeatureDetector.ORB);
    DescriptorExtractor descriptor = DescriptorExtractor.create(DescriptorExtractor.ORB);
    DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE);

    //set up img1 (scene)
    Mat descriptors1 = new Mat();
    MatOfKeyPoint keypoints1 = new MatOfKeyPoint();
    //calculate descriptor for img1
    detector.detect(img1, keypoints1);
    descriptor.compute(img1, keypoints1, descriptors1);

    //set up img2 (template)
    Mat descriptors2 = new Mat();
    MatOfKeyPoint keypoints2 = new MatOfKeyPoint();
    //calculate descriptor for img2
    detector.detect(img2, keypoints2);
    descriptor.compute(img2, keypoints2, descriptors2);

    //match 2 images' descriptors
    MatOfDMatch matches = new MatOfDMatch();
    matcher.match(descriptors1, descriptors2,matches);

    //calculate max and min distances between keypoints
    double max_dist=0;double min_dist=99;

    List<DMatch> matchesList = matches.toList();
    for(int i=0;i<descriptors1.rows();i++)
    {
        double dist = matchesList.get(i).distance;
        if (dist<min_dist) min_dist = dist;
        if (dist>max_dist) max_dist = dist;
    }

    //set up good matches, add matches if close enough
    LinkedList<DMatch> good_matches = new LinkedList<DMatch>();
    MatOfDMatch gm = new MatOfDMatch();
    for (int i=0;i<descriptors2.rows();i++)
    {
        if(matchesList.get(i).distance<3*min_dist)
        {
            good_matches.addLast(matchesList.get(i));
        }
    }
    gm.fromList(good_matches);

    //put keypoints mats into lists
    List<KeyPoint> keypoints1_List = keypoints1.toList();
    List<KeyPoint> keypoints2_List = keypoints2.toList();

    //put keypoints into point2f mats so calib3d can use them to find homography
    LinkedList<Point> objList = new LinkedList<Point>();
    LinkedList<Point> sceneList = new LinkedList<Point>();
    for(int i=0;i<good_matches.size();i++)
    {
        objList.addLast(keypoints2_List.get(good_matches.get(i).queryIdx).pt);
        sceneList.addLast(keypoints1_List.get(good_matches.get(i).trainIdx).pt);
    }
    MatOfPoint2f obj = new MatOfPoint2f();
    MatOfPoint2f scene = new MatOfPoint2f();
    obj.fromList(objList);
    scene.fromList(sceneList);

    //output image
    Mat outputImg = new Mat();
    MatOfByte drawnMatches = new MatOfByte();
    Features2d.drawMatches(img1, keypoints1, img2, keypoints2, gm, outputImg, Scalar.all(-1), Scalar.all(-1), drawnMatches,Features2d.NOT_DRAW_SINGLE_POINTS);

    //run homography on object and scene points
    Mat H = Calib3d.findHomography(obj, scene,Calib3d.RANSAC, 5);
    Mat tmp_corners = new Mat(4,1,CvType.CV_32FC2);
    Mat scene_corners = new Mat(4,1,CvType.CV_32FC2);

    //get corners from object
    tmp_corners.put(0, 0, new double[] {0,0});
    tmp_corners.put(1, 0, new double[] {img2.cols(),0});
    tmp_corners.put(2, 0, new double[] {img2.cols(),img2.rows()});
    tmp_corners.put(3, 0, new double[] {0,img2.rows()});

    Core.perspectiveTransform(tmp_corners,scene_corners, H);


    Core.line(outputImg, new Point(scene_corners.get(0,0)), new Point(scene_corners.get(1,0)), new Scalar(0, 255, 0),4);
    Core.line(outputImg, new Point(scene_corners.get(1,0)), new Point(scene_corners.get(2,0)), new Scalar(0, 255, 0),4);
    Core.line(outputImg, new Point(scene_corners.get(2,0)), new Point(scene_corners.get(3,0)), new Scalar(0, 255, 0),4);
    Core.line(outputImg, new Point(scene_corners.get(3,0)), new Point(scene_corners.get(0,0)), new Scalar(0, 255, 0),4);

The program is able to calculate and display feature points from both images. However, the scene_corners returned are 4 points in a close cluster (small green blob) where they are supposed to represent the 4 corners of the perspective projection of the object onto the scene. I checked double checked to make sure my program is as close to the c++ implementation as possible. What might be causing this?

I checked the homography matrix and it seems the corner coordinates are skewed by 2 very big results from the matrix. Is the homography matrix incorrectly calculated?

I'd appreciate any input, thanks.

Update:

I played about with the filter threshold for good matches and found that 2.75*min_dist seems to work well with this set of images. I can now get good matches with zero outliers. However, the bounding box is still wrong. https://i.sstatic.net/4qjmN.jpg

How do I know what value of threshold to use for best matches and how does the homography relate to them? Why was 3*min_dist used in the example?

Answer 1

Currently I'm also implementing a 2D homography in java and I also found the OpenCV tutorial then your question.

I don't think it'll enhance your results but in the OpenCV tutorial when they compute the min and max distance, they loop with descriptors_object.rows and in your code you do with descriptors1.rows() which is the scene descriptor and not the object descriptor.

Edit: Just also noticed the same with the matcher. For you:

• img1/descriptor1 -> Scene
• img2/descriptor2 -> the object to find

In the tutorial:

matcher.match( descriptors_object, descriptors_scene, matches );

But in your code:

matcher.match(descriptors1, descriptors2,matches);

And Javadoc:

void org.opencv.features2d.DescriptorMatcher.match(Mat queryDescriptors, Mat trainDescriptors, MatOfDMatch matches)

Answer 2

I managed to solve the problem and use homography correctly while investigating index out of bounds errors. It turns out when I added my good matches to my object and scene lists, I swapped round the query and train indices

objList.addLast(keypoints2_List.get(good_matches.get(i).queryIdx).pt);
sceneList.addLast(keypoints1_List.get(good_matches.get(i).trainIdx).pt);

According to this question OpenCV drawMatches -- queryIdx and trainIdx , since I called

matcher.match(descriptors1, descriptors2,matches);

with descriptor1 first then descriptor2, the correct indices should be:

objList.addLast(keypoints2_List.get(good_matches.get(i).trainIdx).pt);
sceneList.addLast(keypoints1_List.get(good_matches.get(i).queryIdx).pt);

where queryIdx refers to keypoints1_List and trainIdx refers to keypoints2_List.

Here is an example result:

https://i.sstatic.net/eakZP.png