How can I improve the quality of the recognition of playing cards?

Question

I am trying to build a classification of playing cards. The goal is that I use an image of a playing card as input and the script then outputs whether it is a playing card and if so, which one.

For this I have written the following comparison method, which I call with the input image and all comparison images, i.e.:

• Input image vs. 2_C
• Input image vs. 2_H
• Input image vs. 2_D
• [...]

Compare method:

def compare_images(img1, img2):
    percent = -1
    detector = cv2.SIFT_create()
   
    kp1, des1 = detector.detectAndCompute(img1,None)
    kp2, des2 = detector.detectAndCompute(img2,None)

    bf = cv2.BFMatcher_create()
    matches = bf.knnMatch(des1,des2,k=2)


    # Apply ratio test
    good = []
    for m,n in matches:
        if m.distance < 0.75*n.distance:
            good.append([m])
            a=len(good)
            percent=(a*100)/len(kp2)

    return percent

It then outputs which playing card matches how well with the input image, but not always reliably..., example:

Input image: 2_C, Output of cv2.drawMatchesKnn(img2, kp2, img1, kp1, matches, None):

Output of percent value:

Match of <2_C> & <2_J>: 197.2972972972973
Match of <2_C> & <2_C>: 28.115942028985508
Match of <2_C> & <A_C>: 27.0
Match of <2_C> & <3_H>: 26.344086021505376
Match of <2_C> & <2_S>: 25.28735632183908
Match of <2_C> & <2_D>: 23.529411764705884
Match of <2_C> & <7_H>: 20.95808383233533
Match of <2_C> & <2_H>: 18.807339449541285
Match of <2_C> & <5_H>: 16.80327868852459
Match of <2_C> & <4_H>: 15.104166666666666
Match of <2_C> & <3_C>: 14.031180400890868
Match of <2_C> & <3_S>: 12.546125461254613
[...]

How can I improve the quality of detection?

Answer 1

It it hard to make reliable recognition with this approach as it may have limited accuracy.

But you could play with feature descriptor and matcher parameters. For example instead of ratio test you can set fixed threshold and count number of good matches:

7ofclubs vs 7ofclubs = 187 good matches:

7ofclubs vs 8ofclubs = 85 good matches:

7ofclubs vs 7ofhearts = 13 good matches:

7ofhearts vs 8ofclubs = 0 good matches:

Complete example:

import cv2

fe = cv2.AKAZE_create(cv2.AKAZE_DESCRIPTOR_KAZE_UPRIGHT, 0, 3, 0.007)

image1 = cv2.imread("card1.png")
image2 = cv2.imread("card2.jpg")

sf = 700. / image1.shape[1]
image1 = cv2.resize(image1, (0, 0), fx=sf, fy=sf)
sf = 700. / image2.shape[1]
image2 = cv2.resize(image2, (0, 0), fx=sf, fy=sf)

kp1, desc1 = fe.detectAndCompute(image1, None)
kp2, desc2 = fe.detectAndCompute(image2, None)

knn_matcher = cv2.DescriptorMatcher_create(cv2.DESCRIPTOR_MATCHER_BRUTEFORCE)
matches = knn_matcher.knnMatch(desc1, desc2, k=2)
good_matches = []

for neighbors in matches:
    for m in neighbors:
        if m.distance < 0.2:
            good_matches.append(m)

print(f"Number of good matches: {len(good_matches)}")

out = cv2.drawMatches(image1, kp1, image2, kp2, good_matches, None)
cv2.imwrite("matches.jpg", out)

Additionally we can use the fact that a card consist of a fixed number of elements: suit (clubs, hearts, etc) and a number on a white background. We can segment these elements:

and locate keypoints that belongs to specific segment and count this keypoint only one per semgent to make sure that the number is matched and the suit is matched.

But from my perspective a more reliable method is to train a neural network.