Reducing misses on template matching (using transparent template)

Question

I'm trying to determine the position of puzzle piece on a puzzle image. I have

1. an image of a puzzle piece (transparent png)
2. an image of a puzzleboard whith a white outline around the correct position

Firstly i extract the contour on image 1. and use it to draw my final template. Then i match the final template to the puzzleboard and save some result images.

Here are some examples of the results i'm getting https://i.sstatic.net/zQxnp.jpg

On some of the images, there's clearly a lot of mismatches and for some of the images, increasing the threshold will hide the correct match and some mismatches will remain.

Any tips or thoughts on optimization would be greatly appreciated!

This is my full code:

full_image = cv2.imread('puzzle_1.jpg')    
piece = cv2.imread('piece_1.png', cv2.IMREAD_UNCHANGED)
partial_image = cv2.cvtColor(piece,cv2.COLOR_BGR2GRAY)

contours, hierarchy = cv2.findContours(partial_image.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
   
template = np.zeros((55, 55, 4), dtype=np.uint8)

cv2.drawContours(template, contours, -1, (255, 255, 255, 255),1)
hh, ww = template.shape[:2]

puzzleP = template[:,:,0:3]
alpha = template[:,:,3]
alpha = cv2.merge([alpha,alpha,alpha])

correlation = cv2.matchTemplate(full_image, puzzleP, cv2.TM_CCORR_NORMED, mask=alpha)

threshhold = 0.98
loc = np.where(correlation >= threshhold)

result = full_image.copy()
for pt in zip(*loc[::-1]):
    cv2.rectangle(result, pt, (pt[0]+ww, pt[1]+hh), (0,0,255), 1)
    print(pt)
    
cv2.imwrite('puzzle_piece.png', puzzleP)
cv2.imwrite('full_image_alpha.png', alpha)
cv2.imwrite('full_image_matches.jpg', result)

EDIT: Here's an example of the two files (piece_1.png and puzzle_1.jpg) (this example has many mismatches) https://i.sstatic.net/HIj7t.jpg

Answer 1

This seems to work fine for me on your given image in Python/OpenCV.

• Read the large and small images
• Convert the small image to gray and threshold to binary
• Get the largest contour of the binary image
• Draw that contour as white on black background to use as the mask
• Extract the BGR channels of the transparent small image as the template
• Do the template matching and get the largest match location
• Draw the match box on a copy of the large image
• Save the results

Large image:

Small image:

import cv2
import numpy as np

# read images
full_image = cv2.imread('puzzle_1.jpg')    
piece = cv2.imread('piece_1.png', cv2.IMREAD_UNCHANGED)

# convert piece to gray and threshold to binary
partial_image = cv2.cvtColor(piece,cv2.COLOR_BGR2GRAY)
partial_image = cv2.threshold(partial_image, 0, 255, cv2.THRESH_BINARY)[1]

# get largest contour from binary image
contours = cv2.findContours(partial_image.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = contours[0] if len(contours) == 2 else contours[1]
big_contour = max(contours, key=cv2.contourArea)

# draw the contour of the piece outline as the mask
mask = np.zeros((55, 55, 3), dtype=np.uint8)
cv2.drawContours(mask, [big_contour], 0, (255,255,255), 1)
hh, ww = mask.shape[:2]

# extract the template from the BGR (no alpha) piece 
template = piece[:,:,0:3]

correlation = cv2.matchTemplate(full_image, template, cv2.TM_CCORR_NORMED, mask=mask)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(correlation)
max_val_ncc = '{:.3f}'.format(max_val)
print("normalize_cross_correlation: " + max_val_ncc)
xx = max_loc[0]
yy = max_loc[1]

print('xmatchloc =',xx,'ymatchloc =',yy)

# draw template bounds and corner intersection in red onto img
result = full_image.copy()
cv2.rectangle(result, (xx, yy), (xx+ww, yy+hh), (0, 0, 255), 1)

# save results
cv2.imwrite('puzzle_template.png', template)
cv2.imwrite('puzzle_mask.png', mask)
cv2.imwrite('full_image_matches.jpg', result)

# show results
cv2.imshow('full_image', full_image)
cv2.imshow('piece', piece)
cv2.imshow('partial_image', partial_image)
cv2.imshow('mask', mask)
cv2.imshow('template', template)
cv2.imshow('result', result)
cv2.waitKey(0)
cv2.destroyAllWindows()

(bgr) Template:

Mask:

Largest Match Location: