Python face recognition slow

Question

I'm trying to build a software that uses face recognition library to detect faces in real time. I tried it using a webcam with promising results and a decently stable frame rate, but when I switched to an .mp4 video, the result was very poor in term of fps. I'm using Python 3.6 with OpenCV and this is the code I'm using:

import face_recognition
import cv2


# Load a sample picture and learn how to recognize it.
totti_image = face_recognition.load_image_file("totti.jpg")
totti_face_encoding = face_recognition.face_encodings(totti_image)[0]

# Create arrays of known face encodings and their names
known_face_encodings = [
    totti_face_encoding
]
known_face_names = [
    "Francesco Totti"
]
def get_faces(frame):
    # Convert the image from BGR color (which OpenCV uses) to RGB color (which face_recognition uses)
    rgb_frame = frame[:, :, ::-1]

    # Find all the faces and face enqcodings in the frame of video
    face_locations = face_recognition.face_locations(rgb_frame)
    face_encodings = face_recognition.face_encodings(rgb_frame, face_locations)

    # Loop through each face in this frame of video
    for (top, right, bottom, left), face_encoding in zip(face_locations, face_encodings):
        # See if the face is a match for the known face(s)
        matches = face_recognition.compare_faces(known_face_encodings, face_encoding, tolerance=0.50)

        name = "Unknown"

        # If a match was found in known_face_encodings, just use the first one.
        if True in matches:
            first_match_index = matches.index(True)
            name = known_face_names[first_match_index]

        # Draw a box around the face
        cv2.rectangle(frame, (left, top), (right, bottom), (0, 0, 255), 2)

        # Draw a label with a name below the face
        cv2.rectangle(frame, (left, bottom - 35), (right, bottom), (0, 0, 255), cv2.FILLED)
        font = cv2.FONT_HERSHEY_DUPLEX
        cv2.putText(frame, name, (left + 6, bottom - 6), font, 1.0, (255, 255, 255), 1)

    return frame

The function "get_faces" is called inside a while loop for every frame, and I'm getting performance around 0.5 fps. If someone has suggestions in order to get better fps in output please let me know, thanks.

EDIT: I used the following example (adapting it to my needs) and everything worked better: link

FINAL CODE:

import face_recognition
import cv2

# Load a sample picture and learn how to recognize it.
image = face_recognition.load_image_file("totti.jpg")
encoding = face_recognition.face_encodings(image)[0]

# Create arrays of known face encodings and their names
known_face_encodings = [
    encoding
]
known_face_names = [
    "Totti",
]

# Initialize some variables
face_locations = []
face_encodings = []
face_names = []

def get_faces(frame):

    # Resize frame of video to 1/10 size for faster face recognition processing
    small_frame = cv2.resize(frame, (0, 0), fx=0.1, fy=0.1)

    # Convert the image from BGR color (which OpenCV uses) to RGB color (which face_recognition uses)
    rgb_small_frame = small_frame[:, :, ::-1]

    # Find all the faces and face encodings in the current frame of video
    face_locations = face_recognition.face_locations(rgb_small_frame)
    face_encodings = face_recognition.face_encodings(rgb_small_frame, face_locations)

    face_names = []
    for face_encoding in face_encodings:
        # See if the face is a match for the known face(s)
        matches = face_recognition.compare_faces(known_face_encodings, face_encoding)
        name = "Person"

        # If a match was found in known_face_encodings, just use the first one.
        if True in matches:
            first_match_index = matches.index(True)
            name = known_face_names[first_match_index]

        face_names.append(name)


    # Display the results
    for (top, right, bottom, left), name in zip(face_locations, face_names):
        # Scale back up face locations since the frame we detected in was scaled to 1/10 size
        top *= 10
        right *= 10
        bottom *= 10
        left *= 10

        # Draw a box around the face
        cv2.rectangle(frame, (left, top), (right, bottom), (0, 0, 255), 2)

        # Draw a label with a name below the face
        cv2.rectangle(frame, (left, bottom - 35), (right, bottom), (0, 0, 255), cv2.FILLED)
        font = cv2.FONT_HERSHEY_DUPLEX
        cv2.putText(frame, name, (left + 6, bottom - 6), font, 1.0, (255, 255, 255), 1)

    return frame

Answer 1

To determine which parts of your script are taking the longest to run, use a profiler. This will output the time to execute for each of your calls, so you can gain a better insight into what parts of your function are suboptimal. See The Python Profilers for examples on how to profile your code.

From the documentation:

SPEEDING UP FACE RECOGNITION

Face recognition can be done in parallel if you have a computer with multiple CPU cores. For example if your system has 4 CPU cores, you can process about 4 times as many images in the same amount of time by using all your CPU cores in parallel. If you are using Python 3.4 or newer, pass in a --cpus <number_of_cpu_cores_to_use> parameter:

face_recognition --cpus 4 ./pictures_of_people_i_know/ ./unknown_pictures/

You can also pass in --cpus -1 to use all CPU cores in your system.

Test the operation on your computer using one and then the maximum number of cores. If this significantly improves execution time, your best course of action is to implement multiprocessing into your own script.

Update 2020-08-05

Looking into this a bit more as it's still getting some attention. If we take a look at the repository, it looks like the CLI just makes some calls that you could script yourself, to get that --cpus agrument into your own code. Specifically, you could use the code here programatically instead of from the command line. Make your own call to either the API in a similar way using multiprocessing, or to def process_images_in_process_pool(images_to_check, number_of_cpus, model):.