Python and OpenCV Image Target Detection and Differentiation

Question

I have been tasked with writing a program to detect and differentiate three "targets" from altitude for my rocketry club. These targets are 3 large tarps for which I have the RGB values.

When I started this project, I overlaid a GoogleEarth image with 3 rectangles using the exact RGB values of the tarps, and my code worked flawlessly. However, when I actually received the tarps and began taking pictures of them on the ground, my code does not recognize the tarps with the RGB color boundaries I prescribed.

I have tried to convert the images to HSV color space, but I just cannot get it to work. I've also thought about using contours - trying to get the program to recognize the 4 straight lines that bound each target. The problem is that these images will be taken outdoors, so I have no control over the ambient lighting conditions.

Does anyone have any ideas as to what colorspace or computer vision method would allow me to identify and differentiate between these targets regardless of outdoor lighting?

Original Image After Processing
Actual Tarps to be Identified

Here is the code:

import cv2
import numpy as np

image = cv2.imread('2000 ft.png', 1)
#hsv_img = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)


#cv2.waitKey(0)
cv2.destroyAllWindows()

# define target strings
targ = ['Target 1 - Blue', 'Target 2 - Yellow', 'Target 3 - Red']
i = 0

# BGR boundaries of colors
boundaries = [
# 0, 32, 91
    ([40, 10, 0], [160, 60, 20]),
# 255, 209, 0
    ([0, 180, 220], [20, 230, 255]),
# 166, 9, 61
    ([40, 0, 150], [80, 30, 185]),
]

# colors for rectangle outlines 
colors = [
          ([91, 32, 0]), ([0, 209, 255]), ([61, 9, 166])
]

# # loop over the boundaries
for (lower, upper) in boundaries:

     # create NumPy arrays from the boundaries
    lower = np.array(lower, dtype = "uint16")
    upper = np.array(upper, dtype = "uint16")

     # find the colors within the specified boundaries and apply
     # the mask
    mask = cv2.inRange(image, lower, upper)
    output = cv2.bitwise_and(image, image, mask = mask)
    # frame threshold
    frame_threshed = cv2.inRange(image, lower, upper)
    imgray = frame_threshed

    # iteratively view masks 
    cv2.imshow('imgray',imgray)
    cv2.waitKey(0)
    cv2.destroyAllWindows()

    ret,thresh = cv2.threshold(frame_threshed,127,255,0)
    contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)

     # Find the index of the largest contour
    areas = [cv2.contourArea(c) for c in contours]
    max_index = np.argmax(areas)
    cont=contours[max_index]

    # putting text and outline rectangles on image
    x,y,w,h = cv2.boundingRect(cont)
    cv2.rectangle(image,(x,y),(x+w,y+h),colors[i],2)
    cv2.putText(image, targ[i], (x-50, y-10), cv2.FONT_HERSHEY_PLAIN, 0.85, (0, 255, 0))
     # cv2.rectangle(image,(x,y),(x+w,y+h),(0,255,0),4)
    cv2.imshow("Show",image)
    cv2.waitKey()
    cv2.destroyAllWindows()
    i += 1

cv2.destroyAllWindows()

I am writing this code in Python, which I have a good amount of experience in, using the OpenCV library which I do not have much experience in. Any help would be greatly appreciated!

Answer 1

I have moved the code to python3/OpenCV3, otherwise it is based on your code

@pandamakes is right in his claim. You need to look for pixels that are close to your target values but you can't assume you'll get something that is very close to that value.

I added a mask for ignoring the boundaries (you get many artifacts around there) and I modified the target values since in real life you are not likely to get pixels with zero value (especially in an aerial image where you have atmospheric reflection)

basically I am looking for a region with value close to the target value and using flood-fill for locating the actual target boundaries

EDIT   moving from the RGB color space to CIELab and using only the ab chromatic channels for added robustness to illumination conditions

import cv2
import numpy as np

image = cv2.imread('tarpsB.jpg', 1)
#convert to CIELab
cielab = cv2.cvtColor(image, cv2.COLOR_BGR2Lab)

# define target strings
targ = ['Target 1 - Blue', 'Target 2 - Yellow', 'Target 3 - Red']
i = 0

# colors = [
#           ([91, 40, 40]), ([40, 209, 255]), ([81, 60, 166])
# ]
# rough conversion of BGR target values to CIELab 
cielab_colors = [
          ([20, 20, -40]), ([80, 0, 90]), ([40, 70, 30])
]

# # loop over the boundaries
height = image.shape[0]
width = image.shape[1]
mask = np.ones(image.shape[0:2])

cv2.circle( mask, (int(width/2), int(height/2)), int(height/2), 0, -1 );
mask = 1-mask
mask = mask.astype('uint8')

#for color in colors:
for cielab_color in cielab_colors:
    diff_img = cielab.astype(float)
    # find the colors within the specified boundaries and apply
    # the mask

    diff_img[:, :, 0] = np.absolute( diff_img[:, :, 0] - 255 * cielab_color[0] / 100 )
    diff_img[:, :, 1] = np.absolute( diff_img[:, :, 1] - (cielab_color[1] + 128) )
    diff_img[:, :, 2] = np.absolute( diff_img[:, :, 2] - (cielab_color[2] + 128) )


    diff_img = ( diff_img[:, :, 1] + diff_img[:, :, 2]) / 2
    diff_img = cv2.GaussianBlur(diff_img, (19, 19), 0)
    minVal, maxVal, minLoc, maxLoc = cv2.minMaxLoc(diff_img, mask)
    min_img = np.array(diff_img/255)

    ff_mask = np.zeros( (height + 2, width + 2), np.uint8)
    cv2.floodFill(image, ff_mask, minLoc, 255, (12, 12, 12), (12, 12, 12), cv2.FLOODFILL_MASK_ONLY );
    ff_mask = ff_mask[1:-1, 1:-1]
    im2, contours, hierarchy = cv2.findContours(ff_mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

    # Find the index of the largest contour
    areas = [cv2.contourArea(c) for c in contours]
    max_index = np.argmax(areas)
    cont=contours[max_index]
    print('target color = {}'.format(image[minLoc[1], minLoc[0], :]))
    # putting text and outline rectangles on image
    x,y,w,h = cv2.boundingRect(cont)
    cv2.rectangle(image,(x,y),(x+w,y+h),colors[i],2)
    cv2.putText(image, targ[i], (x-50, y-10), cv2.FONT_HERSHEY_PLAIN, 0.85, (0, 255, 0))

    cv2.imshow('diff1D',diff_img/255)
    cv2.imshow('ff_mask',ff_mask*255)
    cv2.waitKey(0)
    i += 1

cv2.imshow("Show",image)
cv2.waitKey(0)

cv2.destroyAllWindows()

edit adding output image

Answer 2

The colour from the camera (and indeed, from the rocket) will depend on ambient light, and will unlikely be

# colors for rectangle outlines 
colors = [
          ([91, 32, 0]), ([0, 209, 255]), ([61, 9, 166])
]

You can check that by counting the pixel coordinates of your image, and do a print val.

If you can still change the markers, I would use markers with greater contrast, rather than single colour (e.g. blue square with white border, etc), which would facilitate canny > findContour, and subsequent approxpoly to find the squares.

If changing markers is a no go, then perhaps the best bet is to separate the R, G and B channel, then perform canny > findcontour. I suspect Red and Blue square will go well, but yellow square will go poorly, as it blends into the landscape.