OpenCV: Get correct corners for perspective correction using python

Question

I would like to photograph a LEGO building instruction from different perspectives and determine the perspective of the entire sheet based on the LEGO logo depicted on it.

As a first approach, I detected the square of the logo using a mask in the color space in order to then recognize the edges and use the edges to display the entire image from the correct perspective. (The goal later is to be able to read the ID reliably, regardless of the perspective from which the picture is taken.)

https://i.sstatic.net/U5ZkE.png

filterLegoLogo(image):
  img = image.copy()
  hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

  lower1 = np.array([0, 100, 20])
  upper1 = np.array([10, 255, 255])

  lower2 = np.array([170, 200, 20])
  upper2 = np.array([179, 255, 255])

  lower_mask = cv2.inRange(hsv, lower1, upper1)
  upper_mask = cv2.inRange(hsv, lower2, upper2)

  full_mask = lower_mask + upper_mask;

  return full_mask

Output: https://i.sstatic.net/SL2Op.png

What I want to achieve: https://i.sstatic.net/1DmDj.png

Later I want to transform the recognized area and read out the ID. I use this to transform the image:

image = cv2.imread('images/lego.png')
w, h, _ = np.shape(image)
pts1 = np.float32([[170, 337], [714, 379], [199, 600], [841, 621]])
# Size of transformed image
pts2 = np.float32([[200, 200], [500, 200], [200, 500], [500, 500]])

for val in pts1:
  cv2.circle(image, (int(val[0]), int(val[1])), 5, (0, 255, 0), -1)
M = cv2.getPerspectiveTransform(pts1, pts2)
dst = cv2.warpPerspective(image, M, (h, w))
plt.imshow(dst)
plt.title('Transformed Image')
plt.show()

And as a result I get this: https://i.sstatic.net/4473V.png

The points for pts1 are entered manually. I would like to automate edge detection.

Does anyone have an idea?

Thank you in advance!

Answer 1

Here is one way to do that in Python/OpenCV.

• Read the input
• Threshold using inRange()
• Apply morphology to clean it up
• Get the convex hull
• Get the 4 corner points using approxPolyDP()
• Specify the corresponding output corner points
• Warp the image
• Save the results

Input:

import cv2
import numpy as np

# read image
img = cv2.imread('lego.png')
hh, ww = img.shape[:2]

# threshold on red color
lower=(90,100,170)
upper=(130,150,220)
thresh = cv2.inRange(img, lower, upper)

# apply morphology
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3,3))
morph = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel)
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (7,7))
morph = cv2.morphologyEx(morph, cv2.MORPH_CLOSE, kernel)

# get convex hull
points = np.column_stack(np.where(morph.transpose() > 0))
hull = cv2.convexHull(points)
peri = cv2.arcLength(hull, True)
hullimg = img.copy()
hullimg = cv2.polylines(hullimg, [hull], True, (0,255,0), 1)

# get 4 corners
poly = cv2.approxPolyDP(hull, 0.01 * peri, True)
plist = poly.tolist()
print(plist)
# plist ordered cw from bottom right
cornerimg = img.copy()
cv2.polylines(cornerimg, [np.int32(poly)], True, (0,0,255), 1, cv2.LINE_AA)

# specify control point pairs
# order pts cw from bottom right
inpts = np.float32(poly)
outpts = np.float32([[500, 500], [200, 500], [200, 200], [500, 200]])

# warp image
M = cv2.getPerspectiveTransform(inpts, outpts)
warpimg = cv2.warpPerspective(img, M, (ww,hh))

# save results
cv2.imwrite("lego_thresh.png", thresh)
cv2.imwrite("lego_morph.png", morph)
cv2.imwrite("lego_convexhull.png", hullimg)
cv2.imwrite("lego_corners.png", cornerimg)
cv2.imwrite("lego_warped.png", warpimg)

cv2.imshow('thresh',thresh)
cv2.imshow('morph',morph)
cv2.imshow('hullimg',hullimg)
cv2.imshow('cornerimg',cornerimg)
cv2.imshow('warpimg',warpimg)
cv2.waitKey(0)
cv2.destroyAllWindows()

Threshold Image:

Morphology Cleaned Image:

Convex Hull Image:

4 Corner Image:

Warped Result:

Answer 2

The Mask image gives a good starting point. Extract the contours and find a way to filter out the interesting one (base on size, length, straightness or whatever criterion that works). Then find the point farthest from the center, and the point farthest from the first point (these define the longest diagonal). Next find the two points farthest form the diagonal, on both sides. Now you have the four interesting corners.