CODEWITHSUNDEEP
iosopencvface-detectiondlib
bchen3
bchen3

Reputation: 21

dlib 68 face landmark points are inaccurate when head's roll angle is NOT between -30 to 30 degree

I'm trying to detect faces and extract facial landmarks from iphone camera(BGRA-format) using dlib.

So far, I made it to detect face rects out of AVMetaData and draw 68 dlib landmarks within those bounds on the image. When a detected head's roll-angle (roll-angle data is extracted from AVMetadata as well) is approxiamtely between -30 to 30 degrees, everything works fine. However, if the detected head rolls over 30 degrees, the facial landmarks would not align properly at all. Those 68 points would no longer rotate along the head, saying I'm hold my iphone in landscape orientation mode with home button on the left and the front camera on, the 68-drawn-dots face stays frontal.

I've tested with openGL(offscreen rendering) and openCV for rendering video images, they all generate the same results. The rendering approach seems irrelavant to my problem.

My questions are

  1. How can I properly align those dlib generated 68 points in spite of head roll angle?
  2. Or, is it event possible for dlib to accurately detect any faces that rolls over 30 degrees?
- (NSMutableArray <NSMutableArray <NSValue *> *>*)detecitonOnSampleBuffer:(CMSampleBufferRef)sampleBuffer inRects:(NSArray<NSValue *> *)rects {

    dlib::array2d<dlib::bgr_pixel> img;
    dlib::array2d<dlib::bgr_pixel> img_gray;
    // MARK: magic
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);

    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);
    char *baseBuffer = (char *)CVPixelBufferGetBaseAddress(imageBuffer);

    // set_size expects rows, cols format
    img.set_size(height, width);

    // copy samplebuffer image data into dlib image format
    img.reset();


    long position = 0;
    while (img.move_next()) {
        dlib::bgr_pixel& pixel = img.element();

        // assuming bgra format here
        long bufferLocation = position * 4; //(row * width + column) * 4;
        char b = baseBuffer[bufferLocation];
        char g = baseBuffer[bufferLocation + 1];
        char r = baseBuffer[bufferLocation + 2];
        dlib::bgr_pixel newpixel(b, g, r);
        pixel = newpixel;

        position++;
    }

    // unlock buffer again until we need it again
    CVPixelBufferUnlockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);



    // convert the face bounds list to dlib format
    std::vector<dlib::rectangle> convertedRectangles = [self convertCGRectValueArray:rects bound:CGSizeMake(height, width)];
    dlib::assign_image(img_gray, img);

    NSMutableArray *facesLandmarks = [NSMutableArray arrayWithCapacity:0];
    for (unsigned long j = 0; j < convertedRectangles.size(); ++j) {
        /* original codes start from here */
        dlib::rectangle oneFaceRect = convertedRectangles[j];

        // detect all landmarks
        dlib::full_object_detection shape = predictor(img, oneFaceRect);
        NSMutableArray *eachFaceLandmarks = [NSMutableArray arrayWithCapacity:0];
        for (int i = 0; i < shape.num_parts(); i++) {
            dlib::point p = shape.part(i);
            [eachFaceLandmarks addObject:[NSValue valueWithCGPoint:CGPointMake(p.x(), p.y())]];
        }

        //
        [facesLandmarks addObject:eachFaceLandmarks];
    }

    return facesLandmarks;
}

Upvotes: 2

Views: 1493

Answers (1)

Peter Lee
Peter Lee

Reputation: 381

Honestly, I am not clear with the first question.

Regarding the second one, based on my experience in the past project. We can detect if the face is frontal or inclined by using the line between 2 centers of the eyes. If this line is inclined, we could say that the face needs to be aligned. More detail could be found in the below link where you could find very clear step and steps Face alignment

Upvotes: 1

Related Questions