Which CV algorithm should I choose to detect the corners? (Corner detection, OpenCV)

Question

I'm working on a CV tool for image analysis and use 4 points to form a transformation matrix (map) for image correction (perspective projection - cv.four_point_transform). I have a set of source images obtained from a thermal imager with the presence of distortion. I'm successfully applying a pipeline of standard OpenCV functions. You can see it in the pictures below (Fig. 1). Please note that the main pipeline consists of the following steps:

1. Correction of distortion.
2. Smoothing with a bilateral filter.
3. Threshold.
4. Harris corner detector.
5. Erosion.
6. The weighted average value of a point on a plane over a point cloud after erosion. In the "Target" image you see 4 points, the actual size of 1 pixel (enlarged for clarity).

• Which approach do you think is easier to implement in the future?
• How to properly approach the removal of parasitic points?
• Are there simpler approaches to detection corners?

Unfortunately, I come across cases where the Harris corner detector doesn't cope and doesn't detect obtuse angles. I started testing different approaches such as:

1. Threshold -> Contours -> Approximate Contour -> Points.
2. Threshold -> Canny Edges Detection -> Dilation -> FAST.
3. Threshold -> Canny Edges Detection -> Dilation -> SIFT.
4. Threshold -> Canny Edges Detection -> Dilation -> Probabilistic Hough lines -> Bentley-Ottmann points.

As you can see, some of the approaches can be used, but they form a number of parasitic points. I'm afraid that it will be more difficult to fight them than it seems at first glance (Fig. 2).

Fig. 1 - Successful detection

Fig. 2 - Unsuccessful detection

Answer 1

Not so long ago, Dev Aggarwal (@dev-aggarwal) published an interesting and simple approach for determining a fitted quadrilateral. This approach works for me. See the original post at the link.

The theoretical layout of the algorithm is presented in the article - View Frustum Optimization To Maximize Object’s Image Area.

I duplicate the source code here:

import sympy

def appx_best_fit_ngon(img, n: int = 4) -> list[(int, int)]:
    # convex hull of the input mask
    img_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    contours, _ = cv2.findContours(
        img_gray, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE
    )
    hull = cv2.convexHull(contours[0])
    hull = np.array(hull).reshape((len(hull), 2))

    # to sympy land
    hull = [sympy.Point(*pt) for pt in hull]

    # run until we cut down to n vertices
    while len(hull) > n:
        best_candidate = None

        # for all edges in hull ( <edge_idx_1>, <edge_idx_2> ) ->
        for edge_idx_1 in range(len(hull)):
            edge_idx_2 = (edge_idx_1 + 1) % len(hull)

            adj_idx_1 = (edge_idx_1 - 1) % len(hull)
            adj_idx_2 = (edge_idx_1 + 2) % len(hull)

            edge_pt_1 = sympy.Point(*hull[edge_idx_1])
            edge_pt_2 = sympy.Point(*hull[edge_idx_2])
            adj_pt_1 = sympy.Point(*hull[adj_idx_1])
            adj_pt_2 = sympy.Point(*hull[adj_idx_2])

            subpoly = sympy.Polygon(adj_pt_1, edge_pt_1, edge_pt_2, adj_pt_2)
            angle1 = subpoly.angles[edge_pt_1]
            angle2 = subpoly.angles[edge_pt_2]

            # we need to first make sure that the sum of the interior angles the edge
            # makes with the two adjacent edges is more than 180°
            if sympy.N(angle1 + angle2) <= sympy.pi:
                continue

            # find the new vertex if we delete this edge
            adj_edge_1 = sympy.Line(adj_pt_1, edge_pt_1)
            adj_edge_2 = sympy.Line(edge_pt_2, adj_pt_2)
            intersect = adj_edge_1.intersection(adj_edge_2)[0]

            # the area of the triangle we'll be adding
            area = sympy.N(sympy.Triangle(edge_pt_1, intersect, edge_pt_2).area)
            # should be the lowest
            if best_candidate and best_candidate[1] < area:
                continue

            # delete the edge and add the intersection of adjacent edges to the hull
            better_hull = list(hull)
            better_hull[edge_idx_1] = intersect
            del better_hull[edge_idx_2]
            best_candidate = (better_hull, area)

        if not best_candidate:
            raise ValueError("Could not find the best fit n-gon!")

        hull = best_candidate[0]

    # back to python land
    hull = [(int(x), int(y)) for x, y in hull]

    return hull

Use as follows:

hull = appx_best_fit_ngon(img)

# add lines
for idx in range(len(hull)):
    next_idx = (idx + 1) % len(hull)
    cv2.line(img, hull[idx], hull[next_idx], (0, 255, 0), 1)

# add point markers
for pt in hull:
    cv2.circle(img, pt, 2, (255, 0, 0), 2)

Of the minuses, I see only one additional dependency on the SymPy package, but this is not critical.

Answer 2

You could...

• compensate lens distortion. Then you wouldn't have stray obtuse corners at all, and all the edges would be straight. I believe that's the best path forward.

• see about adjusting the focus of your camera. Yes, I understand it's a thermal camera. They can do that.

• adjust your thresholds for approxPolyDP, so it removes some of the obtuse angles.

• simply filter the corners of the "unsuccessful" detection by angle of corner, discarding sufficiently obtuse ones. Either do this by threshold on the angle, or by ordering all corners and discarding all but the best four. You must be sensitive to the lengths of the edges incident to a corner. If those are short, the corner may not be significant anyway.

Answer 3

I think detecting only the necessary 4 corners robustly for all possible image is difficult goal.

Therefore...

I would consider to employ some corner detection process which make "enough" detection result. Where, "enough" means that at least the necessary 4 corners can be detected (Undetected them is the most serious problem).

In other word, allowing "Some unnecessary corners may be additionally detected". (of cause, "too many" cannot be allowed)

And then, I would consider some RANSAC like approach, for obtaining the transformation matrix.