How can I extract a fingernail from an image with a finger?

Question

Description

I have an image of a finger (with green background) and want to extract the finger nail as a feature. My goal is to describe the contours of the fingernail with functions. However the function part I haven't tried yet and I'm sure I can figure out by myself. I struggle to extract the fingernail and would like to get help by you. You can find images at the end of the post.

What I have done so far:

1. load the image
2. resize the image for less computational effort
3. process it (blur, remove green background, convert into gray scale
4. extract the fingernail from the image (how?)

I tried to do an circle detection or ellipse detection. The circle detection using hough transform doesn't recognize the fingernail. Same for the ellipse detection (apart from the fact that it took 2 mins and that is way too long to wait). Now my question is: Is there an easy way of solving the problem and extract the fingernail?

I also used edge detection / contour detection to extract the fingernail, however it was way too inaccurate and not helpful.

My dream would be to additionally separate the gray/darker part at the beginning of the fingernail, but I did not manage to do that and therefore gave up on this part. However if you know a good and easy way I would love to hear it.

---

The important code fragments:

# imports

# helper functions
def remove_green(img):
    empty_img = np.zeros_like(img)
    RED, GREEN, BLUE = (2, 1, 0)
    reds = img[:, :, RED]
    greens = img[:, :, GREEN]
    blues = img[:, :, BLUE]
    # loop over the image, pixel by pixel
    tmpMask = (greens < 35) | (reds > greens) | (blues > greens)
    img[tmpMask == 0] = (0, 0, 0)  # remove background from original picture
    empty_img[tmpMask] = (255, 255, 255)  # mask with finger in white
    return img, empty_img

# main function
# load and process 
image = cv2.imread(imagePath, 1)  # load
image = cv2.resize(image, None, fx=0.3, fy=0.3)  # resize
image = cv2.GaussianBlur(image, (3, 3), 0)
no_green_image, mask_finger = remove_green(image)  # remove green
gray = cv2.cvtColor(no_green_image, cv2.COLOR_BGR2GRAY)  # gray scalEd
gray_mask_finger = cv2.cvtColor(mask_finger, cv2.COLOR_BGR2GRAY)

# refine edges
kernel = np.ones((5, 5), np.uint8)
gray_mask_finger = cv2.morphologyEx(gray_mask_finger, cv2.MORPH_GRADIENT, kernel)

detect_nail(gray_mask_finger)
# here I struggle

---

Images

The starting image:

Removed green & convert into gray:

Contours:

Answer 1

Go through the project it uses MobileNetV1-FPN-SSD to detect nails in the image with the threshold confidence.

nailtracking

Answer 2

I think the best approach to solve this problem, considering as a semantic/instance segmentation problem, might be using Encoder-Decoder (e.g. U-Net) type architecture since it's quite challenging to solve this problem using conventional image processing based methods. However, i will give it a try. In my approach i followed bellow mentioned steps to detect fingernail region (result is not perfect but you can improve on this):

• Applying threshold on smoothed saturation value to segment ROI
• Gradient calculation using Sobel operator followed by applying threshold to edge region detection (having high gradient value)
• Contour detection and nail region segmentation

image = cv2.imread("finger.jpg")  # load image

hsv_img = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)  # BGR to HSV conversion
hsv_img = cv2.resize(hsv_img, (250, 250))

img_s = hsv_img[:, :, 1]  # Extracting Saturation channel on which we will work

img_s_blur = cv2.GaussianBlur(img_s, (7, 7), 0)  # smoothing before applying  threshold

img_s_binary = cv2.threshold(img_s_blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]  # Thresholding to generate binary image (ROI detection)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5, 5))
img_s_binary = cv2.morphologyEx(img_s_binary, cv2.MORPH_OPEN, kernel, iterations=3)  # reduce some noise

img_croped = cv2.bitwise_and(img_s, img_s_binary) * 2  # ROI only image extraction & contrast enhancement, you can crop this region 

abs_grad_x = cv2.convertScaleAbs(cv2.Sobel(img_croped, cv2.CV_64F, 1, 0, ksize=3))
abs_grad_y = cv2.convertScaleAbs(cv2.Sobel(img_croped, cv2.CV_64F, 0, 1, ksize=3))
grad = cv2.addWeighted(abs_grad_x, .5, abs_grad_y, .5, 0)  # Gradient calculation
grad = cv2.medianBlur(grad, 13)

edges = cv2.threshold(grad, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

cnts = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)  # Contours Detection
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
cnt = None
max_area = 0
for c in cnts:
    area = cv2.contourArea(c)
    if area > max_area:  # Filtering contour
        max_area = area
        cnt = c

cv2.drawContours(hsv_img, [cnt], 0, (0, 255, 0), 3)

Step by step output: