OpenCV Java Smile Detection

Question

I've tried to create a smile detector with source code that I've found on the Internet. It detects face and works pretty well. It uses Haar classifiers, I've found the Haar classifiers for smile recognition and tried it, however it doesn't work. I've tried to use it in the same way that was used to recognize face. Tried the same with eye classifier - and it worked. All classifiers I've found in opencv/data folder, could somebody give me a tip, what could I do more with given code?

import java.io.File;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.MatOfRect;
import org.opencv.core.Point;
import org.opencv.core.Rect;
import org.opencv.core.Scalar;
import org.opencv.highgui.Highgui;
import org.opencv.objdetect.CascadeClassifier;

public class SmileDetector {

public void detectSmile(String filename) {
    System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    System.out.println("\nRunning SmileDetector");

    CascadeClassifier faceDetector = new CascadeClassifier(new File(
            "src/main/resources/haarcascade_frontalface_alt.xml").getAbsolutePath());
    CascadeClassifier smileDetector = new CascadeClassifier(
            new File("src/main/resources/haarcascade_smile.xml").getAbsolutePath());
    Mat image = Highgui.imread(filename);
    MatOfRect faceDetections = new MatOfRect();
    MatOfRect smileDetections = new MatOfRect();
    faceDetector.detectMultiScale(image, faceDetections);

    System.out.println(String.format("Detected %s faces", faceDetections.toArray().length));

    for (Rect rect : faceDetections.toArray()) {
        Core.rectangle(image, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height),
                new Scalar(0, 255, 0));
    }
    Mat face = image.submat(faceDetections.toArray()[0]);
    smileDetector.detectMultiScale(face, smileDetections);

    for (Rect rect : smileDetections.toArray()) {
        Core.rectangle(face, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height),
                new Scalar(0, 255, 0));
    }

    String outputFilename = "ouput.png";
    System.out.println(String.format("Writing %s", outputFilename));
    Highgui.imwrite(outputFilename, image);
    Highgui.imwrite("ee.png", face);
}
}

Answer 1

To answer Vi Matviichuk comment: Yes I was partially able to fix the problem. I've used mouth classifier instead of smile, the name of mouth classifier from opencv samples is haarcascade_mcs_mouth.xml ; then you look for faces, crop them and look for mouths on faces. However it will give you a lot of mouths, so you have to filter them by:

/**
 * Detects face(s) and then for each detects and crops mouth
 * 
 * @param filename path to file on which smile(s) will be detected
 * @return List of Mat objects with cropped mouth pictures.
 */
private ArrayList<Mat> detectMouth(String filename) {
    int i = 0;
    ArrayList<Mat> mouths = new ArrayList<Mat>();
    // reading image in grayscale from the given path
    image = Highgui.imread(filename, Highgui.CV_LOAD_IMAGE_GRAYSCALE);
    MatOfRect faceDetections = new MatOfRect();
    // detecting face(s) on given image and saving them to MatofRect object
    faceDetector.detectMultiScale(image, faceDetections);
    System.out.println(String.format("Detected %s faces", faceDetections.toArray().length));
    MatOfRect mouthDetections = new MatOfRect();
    // detecting mouth(s) on given image and saving them to MatOfRect object
    mouthDetector.detectMultiScale(image, mouthDetections);
    System.out.println(String.format("Detected %s mouths", mouthDetections.toArray().length));
    for (Rect face : faceDetections.toArray()) {
        Mat outFace = image.submat(face);
        // saving cropped face to picture
        Highgui.imwrite("face" + i + ".png", outFace);
        for (Rect mouth : mouthDetections.toArray()) {
            // trying to find right mouth
            // if the mouth is in the lower 2/5 of the face
            // and the lower edge of mouth is above of the face
            // and the horizontal center of the mouth is the enter of the face
            if (mouth.y > face.y + face.height * 3 / 5 && mouth.y + mouth.height < face.y + face.height
                    && Math.abs((mouth.x + mouth.width / 2)) - (face.x + face.width / 2) < face.width / 10) {
                Mat outMouth = image.submat(mouth);
                // resizing mouth to the unified size of trainSize
                Imgproc.resize(outMouth, outMouth, trainSize);
                mouths.add(outMouth);
                // saving mouth to picture 
                Highgui.imwrite("mouth" + i + ".png", outMouth);
                i++;
            }
        }
    }
    return mouths;
}

Then you have to find a smile, I tried to do this with SVM training machine, but I hadn't got enough samples so it wasn't perfect. However, whole code I got can be found here: https://bitbucket.org/cybuch/smile-detector/src/ac8a309454c3467ffd8bc1c34ad95879cb059328/src/main/java/org/cybuch/smiledetector/SmileDetector.java?at=master