Object Detection with OpenCV-Python

Question

I am trying to detect all of the overlapping circle/ellipses shapes in this image all of which have digits on them. I have tried different types of image processing techniques using OpenCV, however I cannot detect the shapes that overlap the tree. I have tried erosion and dilation however it has not helped.

Any pointers on how to go about this would be great. I have attached my code below

original = frame.copy()
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(gray, (3, 3), 0)
canny = cv2.Canny(blurred, 120, 255, 1)
kernel = np.ones((5, 5), np.uint8)
dilate = cv2.dilate(canny, kernel, iterations=1)

# Find contours
cnts = cv2.findContours(dilate, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]

image_number = 0
for c in cnts:
    x, y, w, h = cv2.boundingRect(c)
    cv2.rectangle(frame, (x, y), (x + w, y + h), (36, 255, 12), 2)
    ROI = original[y:y + h, x:x + w]
    cv2.imwrite("ROI_{}.png".format(image_number), ROI)
    image_number += 1

cv2.imshow('canny', canny)
cv2.imshow('image', frame)
cv2.waitKey(0)

Answer 1

Here's a possible solution. I'm assuming that the target blobs (the saucer-like things) are always labeled - that is, they always have a white number inside them. The idea is to create a digits mask, because their size and color seem to be constant. I use the digits as guide to obtain sample pixels of the ellipses. Then, I convert these BGR pixels to HSV, create a binary mask and use that info to threshold and locate the ellipses. Let's check out the code:

# imports:
import cv2
import numpy as np

# image path
path = "D://opencvImages//"
fileName = "4dzfr.png"

# Reading an image in default mode:
inputImage = cv2.imread(path + fileName)
# Deep copy for results:
inputImageCopy = inputImage.copy()

# Convert RGB to grayscale:
grayscaleImage = cv2.cvtColor(inputImage, cv2.COLOR_BGR2GRAY)

# Get binary image via Otsu:
binaryImage = np.where(grayscaleImage >= 200, 255, 0)
# The above operation converted the image to 32-bit float,
# convert back to 8-bit uint
binaryImage = binaryImage.astype(np.uint8)

The first step is to make a mask of the digits. I also created a deep copy of the BGR image. The digits are close to white (That is, an intensity close to 255). I use 200 as threshold and obtain this result:

Now, let's locate these contours on this binary mask. I'm filtering based on aspect ratio, as the digits have a distinct aspect ratio close to 0.70. I'm also filtering contours based on hierarchy - as I'm only interested on external contours (the ones that do not have children). That's because I really don't need contours like the "holes" inside the digit 8:

# Find the contours on the binary image:
contours, hierarchy = cv2.findContours(binaryImage, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)

# Store the sampled pixels here:
sampledPixels = []

# Look for the outer bounding boxes (no children):
for i, c in enumerate(contours):

    # Get the contour bounding rectangle:
    boundRect = cv2.boundingRect(c)

    # Get the dimensions of the bounding rect:
    rectX = boundRect[0]
    rectY = boundRect[1]
    rectWidth = boundRect[2]
    rectHeight = boundRect[3]

    # Compute the aspect ratio:
    aspectRatio = rectWidth / rectHeight

    # Create the filtering threshold value:
    delta = abs(0.7 - aspectRatio)
    epsilon = 0.1

    # Get the hierarchy:
    currentHierarchy = hierarchy[0][i][3]

    # Prepare the list of sampling points (One for the ellipse, one for the circle):
    samplingPoints = [ (rectX - rectWidth, rectY), (rectX, rectY - rectHeight) ]

    # Look for the target contours:
    if delta < epsilon and currentHierarchy == -1:

        # This list will hold both sampling pixels:
        pixelList = []

        # Get sampling pixels from the two locations:
        for s in range(2):

            # Get sampling point:
            sampleX = samplingPoints[s][0]
            sampleY = samplingPoints[s][1]

            # Get sample BGR pixel:
            samplePixel = inputImageCopy[sampleY, sampleX]

            # Store into temp list:
            pixelList.append(samplePixel)

        # convert list to tuple:
        pixelList = tuple(pixelList)
        
        # Save pixel value:
        sampledPixels.append(pixelList)

Ok, there area a couple of things happening in the last snippet of code. We want to sample pixels from both the ellipse and the circle. We will use two sampling locations that are function of each digit's original position. These positions are defined in the samplingPoints tuple. For the ellipse, I'm sampling at a little before the top right position of the digit. For the circle, I'm sapling directly above the top right position - we end up with two pixels for each figure.

You'll notice I'm doing a little bit of data type juggling, converting lists to tuples. I want these pixels stored as a tuple for convenience. If I draw bounding rectangles of the digits, this would be the resulting image:

Now, let's loop through the pixel list, convert them to HSV and create a HSV mask over the original BGR image. The final bounding rectangles of the ellipses are stored in boundingRectangles, additionally I draw results on the deep copy of the original input:

# Final bounding rectangles are stored here:
boundingRectangles = []

# Loop through sampled pixels:
for p in range(len(sampledPixels)):
    # Get current pixel tuple:
    currentPixelTuple = sampledPixels[p]

    # Prepare the HSV mask:
    imageHeight, imageWidth = binaryImage.shape[:2]
    hsvMask = np.zeros((imageHeight, imageWidth), np.uint8)

    # Process the two sampling pixels:
    for m in range(len(currentPixelTuple)):
        # Get current pixel in the list:
        currentPixel = currentPixelTuple[m]

        # Create BGR Mat:
        pixelMat = np.zeros([1, 1, 3], dtype=np.uint8)
        pixelMat[0, 0] = currentPixel

        # Convert the BGR pixel to HSV:
        hsvPixel = cv2.cvtColor(pixelMat, cv2.COLOR_BGR2HSV)
        H = hsvPixel[0][0][0]
        S = hsvPixel[0][0][1]
        V = hsvPixel[0][0][2]

        # Create HSV range for this pixel:
        rangeThreshold = 5
        lowerValues = np.array([H - rangeThreshold, S - rangeThreshold, V - rangeThreshold])
        upperValues = np.array([H + rangeThreshold, S + rangeThreshold, V + rangeThreshold])

        # Create HSV mask:
        hsvImage = cv2.cvtColor(inputImage.copy(), cv2.COLOR_BGR2HSV)
        tempMask = cv2.inRange(hsvImage, lowerValues, upperValues)
        hsvMask = hsvMask + tempMask

First, I create a 1 x 1 Matrix (or Numpy Array) with just a BGR pixel value - the first of two I previously sampled. In this way, I can use cv2.cvtColor to get the corresponding HSV values. Then, I create lower and upper threshold values for the HSV mask. However, the image seems synthetic, and a range-based thresholding could be reduced to a unique tuple. After that, I create the HSV mask using cv2.inRange.

This will yield the HSV mask for the ellipse. After applying the method for the circle we will end up with two HSV masks. Well, I just added the two arrays to combine both masks. At the end you will have something like this, this is the "composite" HSV mask created for the first saucer-like figure:

We can apply a little bit of morphology to join both shapes, just a little closing will do:

    # Set kernel (structuring element) size:
    kernelSize = 3
    # Set morph operation iterations:
    opIterations = 2
    # Get the structuring element:
    morphKernel = cv2.getStructuringElement(cv2.MORPH_RECT, (kernelSize, kernelSize))
    # Perform closing:
    hsvMask = cv2.morphologyEx(hsvMask, cv2.MORPH_CLOSE, morphKernel, None, None, opIterations,cv2.BORDER_REFLECT101)

This is the result:

Nice. Let's continue and get the bounding rectangles of all the shapes. I'm using the boundingRectangles list to store each bounding rectangle, like this:

    # Process current contour:
    currentContour, _ = cv2.findContours(hsvMask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

    for _, c in enumerate(currentContour):
        # Get the contour's bounding rectangle:
        boundRect = cv2.boundingRect(c)

        # Get the dimensions of the bounding rect:
        rectX = boundRect[0]
        rectY = boundRect[1]
        rectWidth = boundRect[2]
        rectHeight = boundRect[3]

        # Store and set bounding rect:
        boundingRectangles.append(boundRect)
        color = (0, 0, 255)
        cv2.rectangle(inputImageCopy, (int(rectX), int(rectY)),
                  (int(rectX + rectWidth), int(rectY + rectHeight)), color, 2)

        cv2.imshow("Objects", inputImageCopy)
        cv2.waitKey(0)

This is the image of the located rectangles once every sampled pixel is processed: