FDDB evaluation code

Question

I'm studying opencv and dlib, for a face detector to use on a university project, and I'm really new at this whole thing of machine learning and computer vision. How can I use the evaluation code from FDDB to evaluate my code for face detection? I'm using dlib's CNN method for detecting faces from images.

import cv2
import dlib

image = cv2.imread('..\\pessoas\\beatles.jpg')

detector = dlib.cnn_face_detection_model_v1("..\\mmods\\mmod_human_face_detector.dat")
detectedFaces = detector(image)

for face in detectedFaces:
    l, t, r, b, c = (int(face.rect.left()), int(face.rect.top()), int(face.rect.right()), int(face.rect.bottom()),
                 face.confidence)
    cv2.rectangle(image, (l, t), (r, b), (255, 0, 0), 2)

cv2.imshow("CNN Detector", image)
cv2.waitKey(0)
cv2.destroyAllWindows()

As you can see the code is pretty simple, but I have to calculate precision, recall, and F1-score to plot the ROC curves and I'm don't know yet how to do it, the readme on the project's github doesn't help.

Answer 1

As to me in ubuntu16, I have to done it by the following steps:

1. Download the fddb original images dataset which you detect face and get detection result.You can download it here.Here is my directory:

2. Join all the images file path to a txt file , and join all the fddb annotations to a txt file. You can download all the files here

As to me I move all the FDDB-FOLD-%d.txt to the directory all_file_path, and then join them to one file by cat * > filePath.txt

Join all the FDDB-fold-%d-ellipseList.txt to one txt by cat *ellipse*.txt > annotFile.txt

Note you may no need to create it, because runEvaluate.pl have do it for you during the running process.

3.Create FDDB evalute exe, download the source code here here And then compile it , you may change the makefile, see the reason here, add

INCS = -I/usr/local/include/opencv

LIBS = -L/usr/local/lib -lopencv_core -lopencv_imgproc -lopencv_highgui
       -lopencv_ml -lopencv_video -lopencv_features2d -lopencv_calib3d 
       -lopencv_objdetect -lopencv_contrib -lopencv_legacy

to the make file.

4. Evaluate, you can use the runEvaluate.pl to evaluate it , but as to me(ubuntu16), I can’t run it directly.

   4.1 change the GUNPLOT path(you should install gnuplot first using it to create ROC image )

4.2 I use rectangle detection model, so I change $detFormat to 0.

my $detFormat = 0; # 0: rectangle, 1: ellipse 2: pixels

4.3 All the images relative path:

my $listFile ="/home/xy/face_sample/evaluation/compareROC/FDDB-folds/filePath.txt";

4.4 All the images annotations

my $annotFile = "/home/xy/face_sample/evaluation/compareROC/FDDB-folds/annotFile.txt";

4.5 The roc file you want to generate(created by evaluate exe):

my $gpFile ="/home/xy/face_sample/evaluation/compareROC/createROC.p";

4.6 You detection file (I will give how to create it latter)

my $detFile ="/home/xy/face_sample/evaluation/compareROC/detDir/fddb_rect_ret1.txt";

It’s content like that:

The ‘runEvaluate.pl’ have some error, change the execute evaluation to the below:

system($evaluateBin, "-a", $annotFile, "-d", $detFile, "-f", $detFormat, "-i", $imDir, "-l", $listFile, "-r", $detDir, "-z", ".jpg");

You can also use command to check it:

xy@xy:~/face_sample/evaluation/compareROC$ ./evaluate \
> -a /home/xy/face_sample/evaluation/compareROC/FDDB-folds/annotFile.txt \
> -d /home/xy/face_sample/evaluation/compareROC/detDir/fddb_rect_ret1.txt \
> -f 0 \
> -i /home/xy/face_sample/evaluation/compareROC/originalPics/ \
> -l /home/xy/face_sample/evaluation/compareROC/FDDB-folds/filePath.txt \
> -r /home/xy/face_sample/evaluation/compareROC/detDir/ \
> -z .jpg

Use python to create fddb evaluation txt file:

def get_img_relative_path():
    """
    :return: ['2002/08/11/big/img_344', '2002/08/02/big/img_473', ......]
    """
    f_name = 'E:/face_rec/face__det_rec_code/face_det/FDDB-folds/all_img_files.txt'
    lst_name = open(f_name).read().split('\n')

    return lst_name

def write_lines_to_txt(lst):
    # lst = ['line1', 'line2', 'line3']
    f_path = 'fddb_rect_ret.txt'
    with open(f_path, 'w') as fp:

        for line in lst:
            fp.write("%s\n" % line)

# For example use opencv to face detection
def detect_face_lst(img):
    """
    :param img: opencv image 
    :return: face rectangles [[x, y, w, h], ..........]
    """
    m_path = 'D:/opencv/sources/data/haarcascades/haarcascade_frontalface_default.xml'
    face_cascade = cv2.CascadeClassifier(m_path)

    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(gray, 1.3, 5)

    return faces


def generate_fddb_ret():
    # The directory from which we get the test images from FDDB
    img_base_dir = 'E:/face_rec/face__det_rec_code/face_det/originalPics/'

    # All the images relative path, like '['2002/08/11/big/img_344', '2002/08/02/big/img_473', ......]'
    lst_img_name = get_img_relative_path()

    # Store detect result, like:
    # ['2002/08/11/big/img_344', '1', '10 10 50 50 1', .............]
    lst_write2_fddb_ret = []

    try:
        for img_name in lst_img_name:
            img_full_name = img_base_dir + img_name + '.jpg'
            img = cv2.imread(img_full_name)

            if img == None:
                print 'error %s not exists, can not generate complete fddb evaluate file' % img_full_name
                return -1

            lst_face_rect = detect_face_lst(img)

            # append img name like '2002/08/11/big/img_344'
            lst_write2_fddb_ret.append(img_name)

            face_num = len(lst_face_rect)
            # append face num, note if no face 0 should be append
            lst_write2_fddb_ret.append(str(face_num))

            if face_num > 0:
                # append each face rectangle x y w h score
                for face_rect in lst_face_rect:
                    # append face rectangle x, y, w, h score
                    # note: opencv hava no confidence so use 1 here
                    s_rect = " ".join(str(item) for item in face_rect) + " 1"
                    lst_write2_fddb_ret.append(s_rect)

    except Exception as e:
        print 'error %s , can not generate complete fddb evaluate file' % e
        return -1

    # Write all the result to txt for FDDB evaluation
    write_lines_to_txt(lst_write2_fddb_ret)

After run the above code you can create FDDB result:

Note: when you create the above txt in windows, if you test it in ubuntu you may get the following errorIncompatible annotation and detection files. See output specifications:

Just copy the content to a new txt file(created in ubuntu) then it solves.

Here is the result:

Some tips:

1. You can see the runEvaluate.pl it's not hard, the above changes may not be needed.You can also change some variable in runEvaluate.pl, like $GNUPLOT, $imDir and so on. add "-z", ".jpg" to system($evaluateBin, "-a", $annotFile, "-d", $detFile, "-f", $detFormat, "-i", $imDir, "-l", $listFile, "-r", $detDir);

   system($evaluateBin, "-a", $annotFile, "-d", $detFile, "-f", $detFormat, "-i", $imDir, "-l", $listFile, "-r", $detDir, "-z", ".jpg");

2. You can also read the evaluate code(mainly the evaluate.cpp which is easy to understand ), so you will have a deep understand of how to evaluate it.

Answer 2

can you explain the step you are in?

You need to download the labelled data from: http://vis-www.cs.umass.edu/fddb/ where it says: Download the database

After that you need to download the result source code: http://vis-www.cs.umass.edu/fddb/results.html

Then you need to modify your program so that the output looks like this:

2002/08/11/big/img_591
1
191 88 164 163 0
2002/08/26/big/img_265
3
52 39 95 95 0
282 59 114 114 0

Where first is the name of the image, then the number of faces in that image, then the coordenates for each face and repeat...

I advice you to build the evaluation on linux since it's a lot easier (at least for me it was).

Hope it helps.