How to interpret matchtemplate output? (openCV, Python)

Question

After reading the docs and searching all over the internet I still do not understand how to interpret the output of the matchTemplate function from openCV.

What I understand:

result = cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED)

I understand that I get kind of a matrix with a matching value for every part in the picture. Each element in this matrix determines how much similarity it shows to the template.

e.g. I can filter all locations that have a matching value below 0.7 with

numpy.where(result >= 0.7)

What I do not understand is how this information is stored in the output I get from the matchTemplate function and how the position of the match can be extracted from the output.

Basically what I wanna do is match several templates to one image and then determine which template matches best to which location (has the max. matchingValue of all applied templates for a location).

My idea is to extract the matching value into a matrix for every template and then compare the matrices (their elements) to one another to find the best match.

Thanks for helping and please correct me where I'm wrong,

Greetings Don

Answer 1

What I do not understand is how this information is stored in the output I get from the matchTemplate function and how the position of the match can be extracted from the output.

This result will return the possibility of each pixel in the image like to top corner pixel of template When you do this

loc =numpy.where(result >= 0.7)

We will filter out the possibility with the above method. We will get the x coordinate, y coordinate for the pixel of the image which the possibility of like to top corner of template greater than 0.7

#(array([202, 203, 203, 203, 204]), array([259, 258, 259, 260, 259]))

Now we get the image locations which are like to top left corner template with the possibility of higher than or equal to 0.7

In our example output, you can see that we can get plenty of matching points based on our threshold. we need to loop them to find each location.

since we know that the loc variable is a tuple with two NumPy array(Y coordinate NumPy array and X coordinate NumPy array), We need to unpack the tuple and reverse the order of arrays to get the actual places of the templates as follow.

Basically what I wanna do is match several templates to one image and then determine which template matches best to which location (has the max. matchingValue of all applied templates for a location).

The problem is, each result is not in the same shape since it depends on the template height and width.

What we can easily do is make all templates to the same shape( which can approximate for all templates) using cv2.resize method

'''
my original  template shapes
img (1256, 1300)
tempate (215, 223)
tempate (217, 204)
tempate (207, 203)
width =220
height = 225
temp  =  cv2.resize(temp,(width,height),cv2.INTER_CUBIC)

This one makes our result in the same shape as below

res = cv2.matchTemplate(gray, temp,cv2.TM_CCOEFF_NORMED)
print(res.shape) #output (1032, 1081)

**My Approach**

1. I set the threshold to 0.7 (your choice) less than this threshold values i will set them into zero ``` res[res

I found the maximum possibility array with all results list. All res are added to results list

maximum_values_array = np.maximum(*results)

3. I found which res has the maximum value for that particular location

maximum_value_contains_array =np.array(results).argmax(axis=0)     

4. I iterate each possibility value greater than zero(threshold>0.7). select the colour based on which array has that maximum value.

for i in range(len(maximum_values_array)):
  for j in range(len(maximum_values_array[i])):
    if maximum_values_array[i][j]>0:
      colour = colours[maximum_value_contains_array[i][j]]
      top_lect = (j,i)
      bottom_right = (j+width, i+height)
      cv2.rectangle(img,top_lect,bottom_right, colour, 2)

Full python code

import cv2
import numpy as np
img  = cv2.imread('test.jpg')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
templates = [cv2.imread('blue_temp.jpg', 0), cv2.imread('yellow_temp.jpg', 0), cv2.imread('red_temp.jpg', 0)]
colours =[(255,0,0), (0,255,0),(0,0,255)]

results =[]
for temp in templates:
  print(temp.shape)
  width =205
  height = 205
  # temp  =  cv2.resize(temp,(width,height),cv2.INTER_CUBIC)
  res = cv2.matchTemplate(gray, temp,cv2.TM_CCOEFF_NORMED)
  res[res<0.9] =0
  results.append(res)
maximum_values_array = np.maximum(*results)
maximum_value_contains_array =np.array(results).argmax(axis=0)     
print(maximum_values_array.shape)
print(maximum_value_contains_array.shape)

for i in range(len(maximum_values_array)):
  for j in range(len(maximum_values_array[i])):
    if maximum_values_array[i][j]>0:
      print(maximum_values_array[i][j])
      colour = colours[maximum_value_contains_array[i][j]]
      top_lect = (j,i)
      bottom_right = (j+width, i+height)
      cv2.rectangle(img,top_lect,bottom_right, colour, 2)
cv2_imshow(img)
cv2.imwrite('output.png', img)


 

Answer 2

It simply slides the template image over the input image (as in 2D convolution) and compares the template and patch of input image under the template image.

If input image is of size (WxH) and template image is of size (wxh), output image will have a size of (W-w+1, H-h+1).

Answer 3

You can use the following code:

min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)

When using cv2.TM_CCOEFF_NORMED max_loc will be the location of the template in your img. And max_val will be the correlation of the match