Unable to increase the region of interest of an image

Question

I am trying to increase the region of interest of an image using the below algorithm.

First, the set of pixels of the exterior border of the ROI is de termined, i.e., pixels that are outside the ROI and are neighbors (using four-neighborhood) to pixels inside it. Then, each pixel value of this set is replaced with the mean value of its neighbors (this time using eight-neighborhood) inside the ROI. Finally, the ROI is expanded by inclusion of this altered set of pixels. This process is repeated and can be seen as artificially increasing the ROI.

The pseudocode is below -

while there are border pixels:
    border_pixels = []

    # find the border pixels
    for each pixel p=(i, j) in image
        if p is not in ROI and ((i+1, j) in ROI or (i-1, j) in ROI or (i, j+1) in ROI or (i, j-1) in ROI) or (i-1,j-1) in ROI or (i+1,j+1) in ROI):
            add p to border_pixels

    # calculate the averages
    for each pixel p in border_pixels:
        color_sum = 0
        count = 0
        for each pixel n in 8-neighborhood of p:
            if n in ROI:
                color_sum += color(n)
                count += 1
        color(p) = color_sum / count

    # update the ROI
    for each pixel p=(i, j) in border_pixels:
        set p to be in ROI

Below is my code

    img = io.imread(path_dir)
    newimg = np.zeros((584, 565,3))
    mask = img == 0
    while(1):
        border_pixels = []
        for i in range(img.shape[0]):
            for j in range(img.shape[1]):
               for k in range(0,3):
                if(i+1<=583 and j+1<=564 and i-1>=0 and j-1>=0):
                    if ((mask[i][j][k]) and ((mask[i+1][j][k]== False) or (mask[i-1][j][k]==False) or (mask[i][j+1][k]==False) or (mask[i][j-1][k]==False) or (mask[i-1][j-1][k] == False) or(mask[i+1][j+1][k]==False))):
                        border_pixels.append([i,j,k])

         if len(border_pixels) == 0:
             break

        for (each_i,each_j,each_k) in border_pixels:
            color_sum = 0
            count = 0
            eight_neighbourhood = [[each_i-1,each_j],[each_i+1,each_j],[each_i,each_j-1],[each_i,each_j+1],[each_i-1,each_j-1],[each_i-1,each_j+1],[each_i+1,each_j-1],[each_i+1,each_j+1]]
            for pix_i,pix_j in eight_neighbourhood:
                if (mask[pix_i][pix_j][each_k] == False):
                    color_sum+=img[pix_i,pix_j,each_k]
                    count+=1
            print(color_sum//count)
            img[each_i][each_j][each_k]=(color_sum//count)

        for (i,j,k) in border_pixels:
            mask[i,j,k] = False
            border_pixels.remove([i,j,k])

    io.imsave("tryout6.png",img)

But it is not doing any change in the image.I am getting the same image as before so I tried plotting the border pixel on a black image of the same dimension for the first iteration and I am getting the below result-

I really don't have any idea where I am doing wrong here.

Answer 1

Here's a solution that I think works as you have requested (although I agree with @Peter Boone that it will take a while). My implementation has a triple loop, but maybe someone else can make it faster!

First, read in the image. With my method, the pixel values are floats between 0 and 1 (rather than integers between 0 and 255).

import urllib
import matplotlib.pyplot as plt
import numpy as np
from skimage.morphology import  binary_dilation, binary_erosion, disk
from skimage.color import rgb2gray
from skimage.filters import threshold_otsu

# create a file-like object from the url
f = urllib.request.urlopen("https://i.sstatic.net/JXxJM.png")

# read the image file in a numpy array
# note that all pixel values are between 0 and 1 in this image
a = plt.imread(f)

Second, add some padding around the edges, and threshold the image. I used Otsu's method, but @Peter Boone's answer works well, too.

# add black padding around image 100 px wide
a = np.pad(a, ((100,100), (100,100), (0,0)), mode = "constant")

# convert to greyscale and perform Otsu's thresholding
grayscale = rgb2gray(a)
global_thresh = threshold_otsu(grayscale)
binary_global1 = grayscale > global_thresh

# define number of pixels to expand the image
num_px_to_expand = 50

The image, binary_global1 is a mask that looks like this:

Since the image is three channels (RGB), I process the channels separately. I noticed that I needed to erode the image by ~5 px because the outside of the image has some unusual colors and patterns.

# process each channel (RGB) separately
for channel in range(a.shape[2]):

    # select a single channel
    one_channel = a[:, :, channel]

    # reset binary_global for the each channel
    binary_global = binary_global1.copy()

    # erode by 5 px to get rid of unusual edges from original image
    binary_global = binary_erosion(binary_global, disk(5))

    # turn everything less than the threshold to 0
    one_channel = one_channel * binary_global

    # update pixels one at a time
    for jj in range(num_px_to_expand):

        # get 1 px ring of to update
        px_to_update = np.logical_xor(binary_dilation(binary_global, disk(1)), 
                                      binary_global)

        # update those pixels with the average of their neighborhood
        x, y = np.where(px_to_update == 1)

        for x, y in zip(x,y):
            # make 3 x 3 px slices
            slices = np.s_[(x-1):(x+2), (y-1):(y+2)]

            # update a single pixel
            one_channel[x, y] = (np.sum(one_channel[slices]*
                                             binary_global[slices]) / 
                                       np.sum(binary_global[slices]))      


        # update original image
        a[:,:, channel] = one_channel

        # increase binary_global by 1 px dilation
        binary_global = binary_dilation(binary_global, disk(1))

When I plot the output, I get something like this:

# plot image
plt.figure(figsize=[10,10])
plt.imshow(a)

Answer 2

This is an interesting idea. You're going to want to use masks and some form of mean ranks to accomplish this. Going pixel by pixel will take you a while, instead you want to use different convolution filters.

If you do something like this:

image = io.imread("roi.jpg")
mask = image[:,:,0] < 30
just_inside = binary_dilation(mask) ^ mask
image[~just_inside] = [0,0,0]

you will have a mask representing just the pixels inside of the ROI. I also set the pixels not in that area to 0,0,0.

Then you can get the pixels just outside of the roi:

just_outside = binary_erosion(mask) ^ mask

Then get the mean bilateral of each channel:

mean_blue = mean_bilateral(image[:,:,0], selem=square(3), s0=1, s1=255)
#etc...

This isn't exactly correct, but I think it should put you in the right direction. I would check out image.sc if you have more general questions about image processing. Let me know if you need more help as this was more general direction than working code.