How to detect an object that blends with the background?

Question

I am a beginner and I am trying to apply an outline to the white remote control on the left that shares the same color with the background.

a = cv2.imread(file_name)
imgGray = cv2.cvtColor(a,cv2.COLOR_BGR2GRAY)

imgGray = cv2.GaussianBlur(imgGray,(11,11),20)

k5 = np.array([[-1,-1,-1],[-1,9,-1],[-1,-1,-1]])
imgGray = cv2.filter2D(imgGray,-1,k5)

cv2.namedWindow("Control")
cv2.createTrackbar("blocksize","Control",33,1000,f)
cv2.createTrackbar("c","Control",3,100,f)
while True:
    strel = cv2.getStructuringElement(cv2.MORPH_CROSS,(3,3))
    blocksize = cv2.getTrackbarPos("blocksize","Control")
    c = cv2.getTrackbarPos("c","Control")

    if blocksize%2==0:
        blocksize += 1
    thrash = cv2.adaptiveThreshold(imgGray,255,cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV,blockSize=blocksize,C=c)
    thrash1 = cv2.adaptiveThreshold(imgGray,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY_INV,blockSize=blocksize,C=c)

    cv2.imshow("mean",thrash)
    cv2.imshow("gaussian",thrash1)
    #r,thrash = cv2.threshold(imgGray,150,255,cv2.THRESH_BINARY_INV)
    key = cv2.waitKey(1000)
    if key == 32 or iter == -1:
         break


edges = cv2.Canny(thrash,100,200)
cv2.imshow('sharpen',sharpen)
cv2.imshow('edges',edges)
cv2.imshow('grey ',imgGray)
cv2.imshow('thrash ',thrash)
cv2.waitKey(0)
circles = cv2.HoughCircles(imgGray,cv2.HOUGH_GRADIENT,1,60,param1=240,param2=50,minRadius=0,maxRadius=0)
contours,_ = cv2.findContours(thrash,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)
putlabel(circles,a,contours)

Those are what I have tried, I have also tried morphological operation such as dilation, erosion, opening and closing too but I am still unable to acquire the result.

Below is my best result but the noise is too severe and the remote controller didn't get fully outlined.

Answer 1

I have thought of a pure image processing approach. But the results are not as accurate as the one depicted by @nathancy

Theory

TLDR; I am using Difference of Gaussians (DoG) which is a 2-stage edge detector.

• Obtain the grayscale image.
• Perform two different blurring operations on it.
• Subtract the blurred images

Blurring operation generally acts as a suppressor of high frequencies. By subtracting the result of two different blurring operations we get a band-pass filter. I would like to quote from this blog "Subtracting one blurred image from the other preserves spatial information that lies between the range of frequencies that are preserved in the two blurred images"

I wrote a simple function that returns the difference of two blurred images:

def dog(img, k1, s1, k2, s2):
    b1 = cv2.GaussianBlur(img,(k1, k1), s1)
    b2 = cv2.GaussianBlur(img,(k2, k2), s2)
    return b1 - b2

Approach

• Obtain grayscale image
• Perform Gaussian blur with different kernel sizes and sigma values
• Subtract the blurred images
• Apply Otsu threshold
• Find contours of sufficiently large area
• Select contours based on their extent

Note: Extent is a property of a contour which is the ration of the contour area to its corresponding bounding rectangle area. Taken from here

Code & Results

img = cv2.imread('path_to_image', cv2.IMREAD_UNCHANGED)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Function to perform Difference of Gaussians
def difference_of_Gaussians(img, k1, s1, k2, s2):
    b1 = cv2.GaussianBlur(img,(k1, k1), s1)
    b2 = cv2.GaussianBlur(img,(k2, k2), s2)
    return b1 - b2

DoG_img = difference_of_Gaussians(gray, 7, 7, 17, 13)

As you can see, it functions as an edge detector. You can vary the kernel sizes (k1, k2) and sigma values (s1, s2)

# Applying Otsu Threshold and finding contours
th = cv2.threshold(DoG_img ,127,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)[1]
contours, hierarchy = cv2.findContours(th, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

# Create copy of original image
img1 = img.copy()

# for each contour above certain area and extent, draw minimum bounding box  
for c in contours:
    area = cv2.contourArea(c)
    if area > 1500:
        x,y,w,h = cv2.boundingRect(c)
        extent = int(area)/(w*h)                
        if extent > 0.6:
            rect = cv2.minAreaRect(c)
            box = cv2.boxPoints(rect)
            box = np.int0(box)
            cv2.drawContours(img1,[box],0,(0,255,0),4)

As you can see, the result is not perfect. The shadows of the objects are also captured during the edge detection process (Difference of Gaussians). You can try varying the parameters to check if the result gets better.

Answer 2

I don't think simple image-processing will be able to isolate an object with the same color as the background. Therefore we have to switch to deep/machine learning. The idea is to remove the background of the image using U-2-Net which will give us a mask of all objects in the foreground then HSV color threshold on white to isolate the object.

Here's the result mask after running it through U-2-Net to remove the background

Bitwise-and to isolate objects

Now we can use traditional image-processing since we can distinguish between the foreground and background. Next we HSV color threshold with a lower/upper color range to isolate white which results in this mask. You can use a HSV color thresholder script to determine the lower/upper ranges.

Now we simply perform a bit of morphological operations to clean up any noise, find contours, and sort by largest contour area. The assumption that the largest contour will be our desired object. Here's the result

Code

import cv2
import numpy as np

# Load image + mask, grayscale, Gaussian blur, Otsu's threshold
image = cv2.imread("1.jpg") # This is the original image
original = image.copy()
mask = cv2.imread("1.png") # This is the mask generated from U-2-Net
gray = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]
bg_removed = cv2.bitwise_and(image, image, mask=thresh)

# HSV color thresholding
hsv = cv2.cvtColor(bg_removed, cv2.COLOR_BGR2HSV)
lower = np.array([0, 0, 0])
upper = np.array([179, 33, 255])
hsv_mask = cv2.inRange(hsv, lower, upper)
isolated = cv2.bitwise_and(bg_removed, bg_removed, mask=hsv_mask)
isolated = cv2.cvtColor(isolated, cv2.COLOR_BGR2GRAY)
isolated = cv2.threshold(isolated, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

# Morph operations to remove small artifacts and noise
open_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3))
opening = cv2.morphologyEx(isolated, cv2.MORPH_OPEN, open_kernel, iterations=1)
close_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
close = cv2.morphologyEx(opening, cv2.MORPH_CLOSE, close_kernel, iterations=1)

# Find contours and sort by largest contour area
cnts = cv2.findContours(close, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)
for c in cnts:
    cv2.drawContours(original, [c], -1, (36,255,12), 3)
    break

cv2.imshow("bg_removed", bg_removed)
cv2.imshow("hsv_mask", hsv_mask)
cv2.imshow('isolated', isolated)
cv2.imshow('original', original)
cv2.waitKey()

If anyone has an approach using simple image processing instead of deep/machine learning, I would love to know how!