How to extract area of an image within a colored border?

Question

I am trying to extract the subsection area of an image based on the border of a colored box on the image (see below).

I want to extract the area of the image within the yellow box..

For reference, I am extracting this image from a PDF using pdfplumber's im.draw_rect function, which requires ImageMagick and Ghostscript. I have looked everywhere I can for a solution to this problem, and while Mark Setchell's answer to the question Python: How to cut out an area with specific color from image (OpenCV, Numpy) has come close, I'm getting some unexpected errors.

Here is what I have tried so far:

import numpy as np
from PIL import Image, ImageFilter
impath = r'Path\to\drawn_p9_image.png'
im = Image.open(impath).convert('RGB')
na = np.array(im)
orig= na.copy()
im = im.filter(ImageFilter.MedianFilter(3))
yellowY, yellowX = np.where(np.all(na==[247,213,83],axis=2))
top, bottom = yellowY[0], yellowY[-1]

But when I run the last line, I get this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: index 0 is out of bounds for axis 0 with size 0

So the NumPy array is not actually capturing the data it is supposed to. When I checked the NumPy array, this is what it output:

>>> na
array([[[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       ...,

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]]], dtype=uint8)

I am not sure why this approach is not working, and am looking for some guidance on how to fix it. I am fine with the yellow boundary being visible in the final cropped image, if that provides an easier solution.

Answer 1

As Mark already pointed out in the comments, the yellow rectangle doesn't have the RGB value of [247, 213, 83]. ImageJ, for example, returns plain yellow [255, 255, 0]. So, using this value might already help.

Nevertheless, to overcome those uncertainties regarding definitive RGB values, maybe also varying across platforms, software, and so on, I'd suggest to use color thresholding using the HSV color space, which also works using Pillow, cf. modes.

You only need to pay attention to the proper value ranges: The hue channel, for example, has values in the range of [0 ... 360] (degree), which are mapped to a full 8-bit, unsigned integer, i.e. to the range of [0 ... 255]. Likewise, saturation and value are mapped from [0 ... 100] (percent) to [0 ... 255].

The remainder is to find proper ranges for hue, saturation, and value (e.g. using some HSV color picker), and NumPy's boolean array indexing to mask yellow-ish areas in the given image.

For the final cropping, you could add some additional border to get rid of the yellow border itself.

Finally, here's some code:

import numpy as np
from PIL import Image


# Convert degree range (0 - 360) to uint8 value range (0 - 255)
def deg_to_uint8(deg):
    return deg / 360 * 255


# Convert percentage range (0 - 100) to uint8 value range (0 - 255)
def perc_to_uint8(perc):
    return perc / 100 * 255


# Open image, and convert to HSV color space for NumPy slicing
img = Image.open('MDRBG.png')
hsv = np.array(img.convert('HSV'))

# Masking color-ish area via NumPy slicing using upper and/or lower
# bounds for hue, saturation, and value
box = hsv[..., 0] > deg_to_uint8(55)        # Hue > 55°
box &= hsv[..., 0] < deg_to_uint8(65)       # Hue < 65°
box &= hsv[..., 1] > perc_to_uint8(80)      # Saturation > 80%
box &= hsv[..., 2] > perc_to_uint8(80)      # Value > 80%

# Find x, y coordinates of masked area; extract first and last elements
xy = np.argwhere(box)
t, b = xy[[0, -1], 0]
l, r = xy[[0, -1], 1]

# For better cropping, maybe add some additional border
bl, bt, br, bb = (3, 3, 3, 3)

# Actual cropping of the image
crop = img.crop((l + bl, t + bt, r - br, b - bb))
crop.save('crop.png')

And, that's the output:

----------------------------------------
System information
----------------------------------------
Platform:      Windows-10-10.0.16299-SP0
Python:        3.9.1
NumPy:         1.20.2
Pillow:        8.1.2
----------------------------------------