Automatically rotating images taken by different cameras

Question

I'm not very good at image processing so this question may have a very simple answer. I'm not even sure that I use the right words to describe my problem.

I'm trying to rotate images taken by different cameras. For example look at the two images below. I have absolutely no information regarding the camera used. I'm trying to rotate the images so the sharpest point of the leaf is always looking up.

I also need a way to mesure to what extend the images were correctly rotated.

I have tried using the code below but I can't seem to get a lot of meaningful information. I think it's because the images are taken with different cameras.

import cv2
import imageio
import matplotlib.pyplot as plt

imgRef = imageio.imread("Ex2/Picture 538.jpg")
imgTest = imageio.imread("Ex2/Capture586_export001_r.JPG")

imgTest_grey = cv2.cvtColor(imgTest, cv2.COLOR_BGR2GRAY)
imgRef_grey = cv2.cvtColor(imgRef, cv2.COLOR_BGR2GRAY)
height, width = imgRef_grey.shape

orb = cv2.ORB_create(1000)
keypoints1, descriptors1 = orb.detectAndCompute(imgRef_grey, None)
keypoints2, descriptors2 = orb.detectAndCompute(imgTest_grey, None)

# Match features.
matcher = cv2.DescriptorMatcher_create(cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING)
matches = matcher.match( descriptors2, descriptors1, None)

# Sort matches by score
matches.sort(key=lambda x: x.distance, reverse=False)

# Remove not so good matches
numGoodMatches = int(len(matches) * 0.3)
matches = matches[:numGoodMatches]

# Draw top matches
imMatches = cv2.drawMatches(imgTest,keypoints2, imgRef, keypoints1, matches, None)
plt.imshow(imMatches)
plt.show()

All help is highly appreciated.

Answer 1

I would say that based on the shape of the leaf (especially the form of the sharpest point), the detection of the correct current angle and the rotation angle would be non-deterministic. Any way id tolerance of 5 deg is ok for you the following is my suggestion.

get original image, make it grayscale, make it blurred and find the biggest contour(the leaf)

gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
blurred = cv.GaussianBlur(gray, (17,17),0)
_, thresh_image = cv.threshold(blurred, 120, 255, cv.THRESH_BINARY_INV)
contours, hierarchy = cv.findContours(thresh_image, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)
c_max = max(contours, key = cv.contourArea)

find the fitted ellipse over the leaf and get the angle of surrounding ellipse. this for sure does not consider the shape of the leaf and as a result, something that from your perspective is +90 is actually -90 (the sharpest point is on the other side). So for example if we detect the angle to be 90 and then rotate is -90 it might happen that the sharpest point is downward, so the hard part would be the detection of the sharpest point being downward and if so rotate it again 180 degrees.

# remove the background so that the rotation is easier (since the aspect ratio of images are not 1)
img_bg_removed = img.copy()
stencil  = np.zeros(img_bg_removed.shape[:-1]).astype(np.uint8)
cv.drawContours(stencil, [c_max], -1, color=(255, 255, 255), thickness=cv.FILLED)
sel      = stencil != 255 # select everything that is not mask_value
img_bg_removed[sel] = (0,0,0)

#rotate the image
img_rotated = rotate_image(img_bg_removed, -1 * angle)

again find the contour and fitting rectangle. cut the fitting rectangle to half and see in which half the proportion of the white point(leaf) to all pixels is less. if the lower half had a lower proportion then the sharp point is there and the leaf should be rotated 180 again.

gray = cv.cvtColor(img_rotated, cv.COLOR_BGR2GRAY)
blurred = cv.GaussianBlur(gray, (17,17),0)
_, thresh_image = cv.threshold(blurred, 50, 255, cv.THRESH_BINARY)
contours, hierarchy = cv.findContours(thresh_image, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)
c_max = max(contours, key = cv.contourArea)
x_rect,y_rect,w_rect,h_rect = cv.boundingRect(c_max)


up_side = thresh_image[y_rect: y_rect+h_rect//2, :][:, x_rect: x_rect + w_rect]
down_side = thresh_image[y_rect+h_rect//2: y_rect+h_rect, :][:, x_rect: x_rect + w_rect]
up_proportion = (up_side[:,:] == 255).sum() / left_side.size
down_proportion = (down_side[:,:] == 255).sum() / right_side.size
if down_proportion < up_proportion:
    img_rotated = rotate_image(img_rotated, -180)

by the way, the rotation function is borrowed from here

PS: this solution can be optimized in a few ways. and one of which is the second Thresholding/Contour-finding can be removed. and simply as we find the surrounding ellipse we find the surrounding rectangle and then as we rotate the leaf we rotate the rectangle too.