What to do with face descriptors of the same person from multiple photos?

Question

Using neural networks for face recognition, the network will generate a "face descriptor" that is, depending on the network of course, often a 128D vector that describes a face ("In general, if two face descriptor vectors have a Euclidean between them less than 0.6 then they are from the same person, otherwise they are from different people." ¹).

I am trying to develop a photographs application that features face recognition, although I do not know how to store the face descriptor of a person. The question is NOT how to serialize them, but when there are hundreds of different photos of the same person, yet each yielding a different face descriptor (because the angle, lighting and etc. are different), how to choose which one to store? Presumably, there should be a way to "improve" the descriptors as more and more photos are processed, without training the neural network.

Thanks!

Answer 1

I asked Adam Geitgey, author of the face_recognition library for Python, and he answered:

There's a few ways to approach this:

1. You could just keep one face encoding per person. But people tend to change over time, etc. So this might not give you the best accuracy with a large image set.

2. You could take all face encodings for each person and train a classifier that decides which person is which for new face encodings. You could use an SVM classifier or even something simple like a linear classifier. This is the strategy that OpenFace uses. The downside is you have to train new classifiers repeatedly as new data is added and verified by the user.

3. A more clever approach is to use a clustering algorithm like Chinese Whispers. The idea is that each face encoding is a node in a graph data structure. The clustering algorithm automatically sorts the nodes so that the closest-distance nodes get clustered and can be assumed to be the same person. There's a dlib example program (written in C++) that demonstrates this.