Merging overlapping rectangle with OpenCV

Question

I'm using two Haar cascade algorithms (frontal and profile) with OpenCV at the same time to improve the face detection.

Unfortunately, the detection is not working correctly and I don't know how to fix it. The return value is 2 (on a picture with 5 faces, normally detected) and all the rectangles have disappeared.

This is the expected result (without the overlapping rectangles):

This is the original picture (and also the result.jpg) if you want to make your own test.

This is the code :

import cv2
import numpy as np

image=cv2.imread("/home/pi/Downloads/test.jpg")
face_cascade=cv2.CascadeClassifier("/home/pi/opencv-3.4.0/data/haarcascades/haarcascade_frontalface_alt.xml")
profil_cascade=cv2.CascadeClassifier("/home/pi/opencv-3.4.0/data/haarcascades/haarcascade_profileface_alt.xml")

gray=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
face=face_cascade.detectMultiScale(gray, 1.06, 5)
profil=profil_cascade.detectMultiScale(gray, 1.1, 5)

combined_array=np.append(face, profil, axis=0)
combined_list=combined_array.tolist()
result=cv2.groupRectangles(combined_list,2)

print("I've found "+str(len(result))+ " face(s)")

for (x,y,w,h) in result[0]:
    cv2.rectangle(image,(x,y),(x+w,y+h),(0,0,255),2)

cv2.imwrite("/home/pi/Download/result.jpg", image)

Answer 1

The Non Maximum Suppression algorithm is used to solve the problem of overlapping detection results. pyimagesearch has a very good article on it and code to get you going in the right direction.

Answer 2

The openCV function groupRectangles requires 3 inputs.

1) rectList : Vector of rectangles

2) groupthreshold : Minimum possible number of rectangles minus 1

3) eps : Relative difference between sides of the rectangles to merge them into a group

Your code:

   result=cv2.groupRectangles(combined_list,1,0.85)

From your code you have set the groupthreshold parameter as 1 which rejects all clusters that has one rectangle. Set this parameter as 0 and you should get the result you wanted.

Solution:

   result=cv2.groupRectangles(combined_list,0,0.85)

Detailed explanation given below: (https://docs.opencv.org/3.4/d5/d54/group__objdetect.html#ga3dba897ade8aa8227edda66508e16ab9)

Answer 3

After a lot of research, i've partially solved the problem.

I've changed the Threshold and EPS of result=cv2.groupRectangles and I've also make a subtraction in print function between the number of total faces detected (in combined_list) and the number of overlapping detection (return by result)

Here's the new code :

import cv2
import numpy as np

image=cv2.imread("/home/pi/Downloads/test.jpg")
face_cascade=cv2.CascadeClassifier("/home/pi/opencv-3.4.0/data/haarcascades/haarcascade_frontalface_alt.xml")
profil_cascade=cv2.CascadeClassifier("/home/pi/opencv-3.4.0/data/haarcascades/haarcascade_profileface_alt.xml")

gray=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
face=face_cascade.detectMultiScale(gray, 1.06, 5)
profil=profil_cascade.detectMultiScale(gray, 1.1, 5)

combined_array=np.append(face, profil, axis=0)
combined_list=combined_array.tolist()
result=cv2.groupRectangles(combined_list,1,0.85)

print("I've found "+str(len(combined_list)-str(len(result[1]))+ " face(s)")

for (x,y,w,h) in result[0]:
    cv2.rectangle(image,(x,y),(x+w,y+h),(0,0,255),2)

cv2.imwrite("/home/pi/Download/result.jpg", image)

The overlapping rectangles have disappear... but also the other rectangles !

Finally, the program give me the correct number of face (5) and redraw overlapping rectangle (that's good news), but the non-overlapping rectangles have disappear...

I've tried to solve this by using a np.subtract between the coordinate from combined_list and result, then draw the missing rectangle with for (x,y,w,h) in np.subtract[0] but it didn't work. The reason is that the coordinate point of overlapping rectangle are directly recalculate, so i can't make the subtraction with the original coordinate points

If someone has a idea to solve this issue, don't hesitate :)