Stable icon recognition by opencv python

Question

I have the following issue: I need to detect icons in an image reliably. The image also contains text, and the icons come in various sizes.

Currently, I'm using Python with the cv2 library for this task. However, unfortunately, the current contour detection algorithm using cv2.findContours isn't very reliable. Here's how I'm currently doing it:

gray = cv2.cvtColor(self.image, cv2.COLOR_BGR2GRAY)
binary = cv2.adaptiveThreshold(self.gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 17, 1)
contours, _ = cv2.findContours(self.binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

Then follows contour filtering, merging filtered contours, and filtering again.

However, this method has proven to be unreliable.

I've also tried using getStructuringElement, but it gives unreliable results for icons on a white background.

I can't disclose real input data, but I used the Amazon logo and created an example demonstrating the issue.

For colored icons, when using contours, I often get two or three icons of incorrect sizes, and merging them loses the precise size. For icons on a white background, the approach with getStructuringElement doesn't detect the boundary well.

My questions: What do you suggest? My ideas:

1. Train a Haar cascade. Would it help much?
2. Fine-tune parameters of one of the current methods.
3. Any other methods/libraries.

I'm open to any suggestions, or let me know if anyone has experience solving such a problem.

Img: https://i.sstatic.net/bZYrnCTU.jpg

PS: For another people, who's maybe be interested in more or less the same task

Answer from @Tino-d really great, Bot you may get and icon, which more or less would be divided into big amount of simple polygons

For example(After canny processing)

What you can do its just draw contours with bigger thickness(2 for example), and then refounding contours on new image

contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
cv2.drawContours(edges, contours, -1, (255, 255, 255), 2)
contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

Seems like more or less super-easy and very fast merging contours algo)

Answer 1

Triggered by the comment of OP to another question which uses calculation of a bounding boxes area value to filter icons out of other detected objects here another approach helping to obtain correct outlines of the icons also in case of the middle icon in provided image.

The core idea for this kind of icon detection is filtering on the size and shape of the detected object. This allows more generous threshold to capture all the details of an icon almost no matter how slight the outer contour may be. With a threshold of 250 we get:

which contour detection result looks like:

With this set of bounding boxes filtering on large areas will fail to provide correct results, so it is necessary to analyse the shape and size of the contours and limit it to the expected size and shape of the icons.

Below how the result looks like if choosing the range for width and height of the bounding boxes of found contours between 105 and 120:

and the code for obtaining this result is:

import cv2
import numpy as np
srcImg = cv2.imread("iconDetection.jpg")

grayImg = cv2.cvtColor(srcImg, cv2.COLOR_BGR2GRAY)
ret, bwImg  = cv2.threshold( grayImg, 250, 255, cv2.THRESH_BINARY_INV)
cv2.imshow('Binary', bwImg) ; cv2.waitKey(0); cv2.destroyAllWindows() 

Contours , _    = cv2.findContours(bwImg.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

BBoxes=[] # Bounding Boxes of found reaction fields on photographed strips 
for contour in Contours:
    x, y, w, h = cv2.boundingRect(contour)
    #print(f"{x=}{y=}{w=}{h=}")
    if w in range(105, 120) and h in range(105,120):
        BBoxes += [ (x, y, w, h) ] # include only large areas 
        cv2.rectangle(srcImg, (x, y), (x + w, y + h), (0, 255, 0), 3)
cv2.imshow('Bounding Rectangles', srcImg) ; cv2.waitKey(0); cv2.destroyAllWindows() 

Notice that if you change the approach to usage of Canny edge detection it will increase the computation load and need a change of the range of height/width to ( 100 , 120 ), else the amazon logo will be missing in the resulting found bounding boxes.

Below how it looks like if you replace:

ret, bwImg  = cv2.threshold( grayImg, 250, 255, cv2.THRESH_BINARY_INV)

with

bwImg = cv2.Canny(grayImg, 5, 20)

without changing the size range from (105 , 120 ) to ( 100, 120 ):

Answer 2

To increase the reliability use the contour data for checking the area of the contour. The icons seem to be larger in area as the text so you can set a threshold for the area to filter them out using minAreaRectangle = cv2.minAreaRect(contour) :

import cv2
import numpy as np
srcImg = cv2.imread("iconDetection.jpg")

grayImg = cv2.cvtColor(srcImg, cv2.COLOR_BGR2GRAY)
ret, bwImg  = cv2.threshold( grayImg, 200, 255, cv2.THRESH_BINARY_INV)
cv2.imshow('Binary', bwImg) ; cv2.waitKey(0); cv2.destroyAllWindows() 

Contours , _    = cv2.findContours(bwImg.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

BBoxes=[] 
for contour in Contours:
    minAreaRectangle = cv2.minAreaRect(contour)
    bBox = cv2.boxPoints(minAreaRectangle)
    if cv2.contourArea(bBox)  > 60 : # exclude small objects  
        BBoxes += [ np.intp(bBox) ] # include only large areas 

for bBox in BBoxes: 
    tgtImg = cv2.drawContours(srcImg ,[bBox],0,(0,255,0),3)

cv2.imshow('Bounding Rectangles', tgtImg) ; cv2.waitKey(0); cv2.destroyAllWindows() 

gives:

Answer 3

I think edge detection can work quite well here, I did a small example which worked for now:

im = cv2.imread("logos.jpg")
imGray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(imGray,5, 20)

This gave the following result:

After this, detecting contours and filtering by area will work quite nicely, as the squares of the logos seem to all be the same size:

contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
sortedContours = sorted(contours, key = cv2.contourArea, reverse = True)
for c in sortedContours:
    print(cv2.contourArea(c))

We see that indeed the three biggest contours by area all have around 10500 pixels:

10787.0
10715.0
10128.0
7391.5
4555.5
3539.0
3420.0
.
.
.

Fill those first three contours:

im1 = cv2.drawContours(im.copy(), sortedContours, 2, (255,0,0), -1)
im2 = cv2.drawContours(im.copy(), sortedContours, 1, (0,255,0), -1)
im3 = cv2.drawContours(im.copy(), sortedContours, 0, (0,0,255), -1)

And this is what you will get:

I assume what you want is a boolean mask to be able to get those pixels. So something like

mask = np.zeros_like(imGray)
mask = cv2.drawContours(mask, sortedContours, 2, 1, -1)
firstLogo = cv2.bitwise_and(im, im, mask = mask)

Can do the job. You can automate this quite easily by filtering the contours, I'm just nudging a POC to you.

E: forgive the weird colours, forgot to convert to RGB before imshow with Matplotlib...