Find contours of rectangular objects touching each other

Question

Here is grayscale uint8 image I'm working with: source grayscale image. This image is a result of stitching 6 different colorized depth images into one. There are 3 rectangular objects in the image, and my goal is to find edges of these objects. Obviously, I have no problem to find external edges of objects. But, separating objects from each other is a big pain.

Desired rectangles in image: Input image as numpy array: https://drive.google.com/file/d/1uN9R4MgVQBzjJuMhcqWMUAhWDJCatHSf/view?usp=sharing

• First of all I was trying to threshold binarize the image, following with some erosion + dilation processing to distinguish all three objects from each other. Then contours + minAreaRect would give me necessary result. This option isn't robust enough, because objects in the scene can be so close to each other, that edge between them has the same depth as roughness of the object surfaces. So important edges can be "blended" with object surfaces deviations. Consequently, sometimes, I'm getting two objects united in one object.
• Using canny edge detection with automatically calculated coefficients (from picture median) catches all unnecessary brightness changes together with edges. Canny with manually adjusted coefficients works better, but it doesn't give closed edge result + it is not reliable (must be manually tweaked each time).
• Another thing I tried - adjusting brightness of image nonlinearly (power-law transformation) - to increase brightness of objects surfaces leaving dark edge cavities unchanged.

p = 0.2; c = (input_image.max()) / (input_image.max()**(p)); output_image = (c*blur_gray.astype(np.float)**(p)).astype(np.uint8)

Here is a result: brightness adjusted image. Threshold binarizing of this image give better results in terms of edges. I tried canny and Laplacian edge detection, but obtained results give disconnected parts of contour with some noise in object surface areas: binarized result of Laplacian filtering. Next step, in my mind, must be some kind of edge estimation/restoration algorithm. I tried Hough transform to get edge lines, but it didn't give any intelligible result.

---

It seems to me that I just go around in circles without achieving any intelligible result. So I request help. Probably my approach is fundamentally wrong, or I am missing something due to the fact that I do not have sufficient knowledge. Any ideas or suggestions?

P.S. After posting this, I'll continue, and will try to implement wateshed segmentation algorithm, may be it would work.

Answer 1

I tried to come up with a method to emphasize the vertical and horizontal lines separating the shapes.

I started by thresholding the original image (from numpy) and just used a [0, 10] range that seemed reasonable.

I ran a vertical and horizontal line kernel over the image to generate two masks

Vertical Kernel

Horizontal Kernel

I combined the two masks so that we'd have both of the lines separating the boxes

Now we can use findContours to find the boxes. I filtered out small contours to get just the 3 rectangles and used a 4-sided approximation to try and get just their sides.

import cv2
import numpy as np
import random

# approx n-sided shape
def approxSides(contour, numSides, step_size):
    # approx until numSides points
    num_points = 999999;
    percent = step_size;
    while num_points >= numSides:
        # get number of points
        epsilon = percent * cv2.arcLength(contour, True);
        approx = cv2.approxPolyDP(contour, epsilon, True);
        num_points = len(approx);

        # increment
        percent += step_size;

    # step back and get the points
    # there could be more than numSides points if our step size misses it
    percent -= step_size * 2;
    epsilon = percent * cv2.arcLength(contour, True);
    approx = cv2.approxPolyDP(contour, epsilon, True);
    return approx;

# convolve
def conv(mask, kernel, size, half):
    # get res
    h,w = mask.shape[:2];

    # loop
    nmask = np.zeros_like(mask);
    for y in range(half, h - half):
        print("Y: " + str(y) + " || " + str(h));
        for x in range(half, w - half):
            total = np.sum(np.multiply(mask[y-half:y+half+1, x-half:x+half+1], kernel));
            total /= 255;
            if total > half:
                nmask[y][x] = 255;
            else:
                nmask[y][x] = 0;
    return nmask;

# load numpy array
img = np.load("output_data.npy");
mask = cv2.inRange(img, 0, 10);

# resize
h,w = mask.shape[:2];
scale = 0.25;
h = int(h*scale);
w = int(w*scale);
mask = cv2.resize(mask, (w,h));

# use a line filter
size = 31; # size / 2 is max bridge size
half = int(size/2);
vKernel = np.zeros((size,size), np.float32);
for a in range(size):
    vKernel[a][half] = 1/size;
hKernel = np.zeros((size,size), np.float32);
for a in range(size):
    hKernel[half][a] = 1/size;

# run filters
vmask = cv2.filter2D(mask, -1, vKernel);
vmask = cv2.inRange(vmask, (half * 255 / size), 255);
hmask = cv2.filter2D(mask, -1, hKernel);
hmask = cv2.inRange(hmask, (half * 255 / size), 255);
combined = cv2.bitwise_or(vmask, hmask);

# contours OpenCV3.4, if you're using OpenCV 2 or 4, it returns (contours, _)
combined = cv2.bitwise_not(combined);
_, contours, _ = cv2.findContours(combined, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE);

# filter out small contours
cutoff_size = 1000;
big_cons = [];
for con in contours:
    area = cv2.contourArea(con);
    if area > cutoff_size:
        big_cons.append(con);

# do approx for 4-sided shape
colored = cv2.cvtColor(combined, cv2.COLOR_GRAY2BGR);
four_sides = [];
for con in big_cons:
    approx = approxSides(con, 4, 0.01);
    color = [random.randint(0,255) for a in range(3)];
    cv2.drawContours(colored, [approx], -1, color, 2);
    four_sides.append(approx); # not used for anything

# show
cv2.imshow("Image", img);
cv2.imshow("mask", mask);
cv2.imshow("vmask", vmask);
cv2.imshow("hmask", hmask);
cv2.imshow("combined", combined);
cv2.imshow("Color", colored);
cv2.waitKey(0);