Facial recognition systems improve recognition of masked faces

Facial recognition algorithms are improving to recognize faces in masks, according to data released Tuesday by the National Institute for Standards and Technology (NIST). Based on independent testing of more than 150 separate facial recognition algorithms, the new report suggests that masks may not be as big of an issue for facial recognition systems as initially thought.

Vendors voluntarily submit their facial recognition algorithms to NIST for testing as part of the Facial Recognition Vendor Test (FRVT). The institute publishes the results of these tests on an ongoing basis as each algorithm is submitted. When NIST first looked at the effect of masks on facial recognition in July, it found that the algorithms weren’t very good at identifying faces with masks. Not surprisingly, it is more difficult to recognize a face when the nose and mouth are covered.

NIST reports focus on false non-match rates (FNMRs), a measure of the number of matching faces that slip through the algorithm without raising an alert. In July, the error rate of some algorithms reached between 5 and 50% when faced with images of masked people.

Image: NIST

But the pandemic has given developers plenty of time to focus on the mask issue, and NIST data shows facial recognition algorithms are working better and better with masked faces. Without masks, the best algorithms have a mismatch rate of around 0.3% – but that number still increases to 5% when high coverage masks are worn.

“Although a few pre-pandemic algorithms remain among the most accurate on masked photos, some developers have submitted algorithms after the pandemic, showing significantly improved accuracy and are now among the most accurate in our test,” the report said.

NIST’s public ranking for facial recognition tests confirms this claim. Eight different algorithms now have false mismatch rates below 0.05%. Six of those eight were submitted to NIST after the first report was released in July.

The authors note a number of limitations to the study. In particular, while the tests relied on photos of real visa holders and real photos of border crossing, they did not use real images of masked faces. For convenience, the NIST researchers instead applied masks digitally to ensure consistency in the sample. As a result, “we have not been able to pursue an exhaustive simulation of the endless variations in color, design, shape, texture, banding and ways in which masks may be worn,” the report notes. The digital mask was a blue surgical mask covering the entire width of the face, but the testers noted that performance varied greatly depending on the height of the mask on the face.

The United States uses facial recognition at land and air borders, matching travelers to their visa or passport photos as part of the biometric exit program. NIST data is drawn specifically from visa holders who have little privacy rights over biometric information collected during the immigration process.