# FindFaces: how to improve the results of the face recognition feature

In Mathematica 9, the FindFaces command offers an easy way to detect faces. In some experiments, I've been getting reasonable results, but I'd like to know if there are ways to improve the success rate.

As with the online help example, I thought the 1927 Solvay Conference photograph was a good place to start. You can find it on the Wikimedia here, at 3000 by 2171 pixels.

solvay = Import["http://upload.wikimedia.org/wikipedia/commons/6/6e/Solvay_conference_1927.jpg"];
faces = FindFaces[solvay]


which returns a list of rectangles:

{{{81.5, 1105.5}, {262.5, 1286.5}}, {{133.5, 1038.5}, {231.5, 1136.5}}, ...


Using these to chop out some passport photographs:

passports = ImageTrim[solvay, #] & /@ faces


gives this:

I can get rid of the images that are too large by using the optional minimum and maximum sizes:

faces1 = FindFaces[solvay, {85, 130}];
passports1 = ImageTrim[solvay, #] & /@ faces1


But it's still not perfect. (The online help example seems to miss more people than I do, but there are no false positives, unlike my interesting collection of masonry people.

How would I improve these results? Is Mathematica's built-in function capable of better results, or is it just, well, a toy?

-
I love the first one – Dr. belisarius Dec 20 '12 at 10:39
@belisarius angry physicist? :) – cormullion Dec 20 '12 at 10:41
The false positives appear to be very sensitive to denoising. If you run, say, a TotalVariationFilter on the image, many of the false positives go away and new false positive appear (the new ones are all different for Method -> "Gaussian", "Laplacian", or "Poisson"). The obvious solution: apply a bunch of different filters to the image, do FindFaces on the results, and keep the faces that appear in all of them. :) – Rahul Dec 20 '12 at 11:05
Mr. Angry Physicist never goes away though. I think he looks like Freud. – Rahul Dec 20 '12 at 11:08
For those that might enjoy it: i.imgur.com/M5tCc.jpg – R. M. Dec 20 '12 at 19:40

The built-in function is certainly capable of different results, using the hidden options "TrainingFile" and "ScaleDecreaseFraction".

Under the hood, Mathematica uses OpenCV for face detection. The algorithm is controlled by information contained in an XML file, the one supplied with Mathematica is located here:

FileNameJoin[{$InstallationDirectory, "SystemFiles", "Data", "Haarcacades", "frontalface.xml"}]  You can download other "trained classifiers" from the OpenCV GitHub repository here. As well as alternative classifiers for faces, there are also files for detecting various other features, such as eyes, ears, upperbody and so on. These files can be supplied to FindFaces with the "TrainingFile" option. Here are a couple of examples (I have copied the XML files from GitHub into my $UserDocumentsDirectory):

findfeatures[image_, file_, opts___] := Module[{features},
features = FindFaces[image,
{"TrainingFile" -> FileNameJoin[{\$UserDocumentsDirectory, file}], opts}];
Show[image, Graphics[{EdgeForm[{Cyan, Thick}], Opacity[0], Rectangle @@@ features}]]]


Find Lena's eyes:

findfeatures[ExampleData[{"TestImage", "Lena"}], "haarcascade_mcs_eyepair_big.xml"]


Find Girl3's nose:

findfeatures[ExampleData[{"TestImage", "Girl3"}], "haarcascade_mcs_nose.xml"]


The results often include false positives, they can sometimes be improved with the "ScaleDecreaseFraction" option, which takes a real number between 0 and 1. Unfortunately I don't understand exactly what this option does, so it is a case of trial and error to find a good value.

For example, finding Tiffany's eyes worked well with a value of 0.7 (note that I'm using a classifier for individual eyes here, as opposed to the "eye pair" I used on Lena):

findfeatures[ExampleData[{"TestImage", "Tiffany"}], "haarcascade_eye.xml",
"ScaleDecreaseFraction" -> 0.7]


The best I could do with the Solvay Conference image was this:

findfeatures[solvay, "haarcascade_frontalface_alt2.xml",
"ScaleDecreaseFraction" -> 0.2]


There are no false positives, but the algorithm has failed to find Lorentz (possibly because of the beard?)

I suspect, as Rahul Narain commented, to get perfect results you would have to run the algorithm several times with different parameters and/or image pre-processing, and then do some analysis of the results.

-
Excellent spelunking! – R. M. Sep 13 '13 at 21:19
Fascinating! There's much functionality hidden away, for some reason. – cormullion Sep 13 '13 at 21:21
@cormullion, yep. I'm sure WRI have their reasons for it, but it can be quite frustrating sometimes. – Simon Woods Sep 13 '13 at 21:28
Lorentz is surely contracted. +1 of course. – Dr. belisarius Oct 24 '14 at 22:05
@bobthechemist, here is the complete course: Use this, plus trial and error ;-) – Simon Woods Oct 25 '14 at 15:33