Makeup challenges automated face recognition systems

The accuracy of face recognition systems has rapidly increased over the past decade, primarily due to robust feature representation and matching techniques. These methods are resilient to variations in facial pose, expression, and illumination.¹ However, there are additional factors that compromise the use of these systems in security applications. These include aging—a natural biological change—and plastic surgery²—a medically induced change—both known to reduce accuracy (see Figure 1).

Figure 1. Challenges in face recognition: images of actress Jennifer Grey demonstrating variations due to pose, illumination, expression, and alterations through plastic surgery, aging and makeup. (Images obtained from www.imdb.com, www.nydailynews.com, and YouTube).

Another factor is non-permanent makeup that can alter the perceived shape and texture of a face. Makeup can be used strategically for spoofing or obfuscation (see Figure 2). In spoofing, an adversary modifies his facial appearance to resemble another individual; in obfuscation, he makes alterations to evade recognition by a biometric system. We have considered the possibility of unintentional obfuscation through the use of facial makeup.

Figure 2. The impact of makeup. (a) The subject before makeup application. (b-f) After application. (Images obtained from YouTube).

Makeup is a cost-efficient and socially acceptable way to substantially change the appearance of the face. Specifically, it can alter the apparent shape by accentuating contouring techniques (see Figure 3a). It can change the appearance of skin quality and color (see Figure 3b), alter the perceived nose shape and size, enhance or reduce the size of the mouth (see Figure 3c), and modify contrast by adding color. Makeup may alter the apparent form, shade, and location of the eyebrows (see Figure 5a, b), and the perceived shape, size, and contrast of the eyes (see Figure 4e–h). Other effects include concealing dark circles under the eyes (see Figure 5c, d). In addition, makeup can camouflage wrinkles, birth moles, scars, and tattoos. A vast cosmetics market attempts to improve facial aesthetics while projecting good health.

Figure 3. Makeup effects that significantly alter facial appearance. (a) Contouring. (b) Concealing. (c) Enhancing. (Images obtained from YouTube).

Figure 4. Four samples from the YouTube MakeUp (YMU) tutorials dataset.⁵Each subject is shown before applying makeup (a, c, e, g) and after (b, d, f, h). Two subjects have subtle makeup (b, d), and two heavy makeup (f, h). (Images obtained from YouTube).

Figure 5. How makeup alters the appearance of the eyes and eyebrows. (Images obtained from YouTube).

The impact of makeup on human ability to recognize faces has been studied by Ueda and Koyama.³ These authors observed that the application of light makeup slightly increases human recognizability, whereas heavy makeup significantly decreases it. Light makeup accentuates distinctive facial characteristics, thereby facilitating face recognition, while heavy makeup increases the bilateral size and symmetry of the eyes and lips, leading to a decreased characteristic distinctiveness.⁴ Light makeup is not easily perceived, since the applied colors are natural skin, lip, and eye colors (see Figure 4b, d). Heavy makeup is clearly visible, for example, red or dark lips, and strongly accentuated eyes (see Figure 4f, h). The notion of light or heavy makeup does not necessarily relate to the number of products used but rather to the difference in facial appearance before and after application (see Figure 3b).

We studied the impact of cosmetic makeup on automated face recognition systems. In an earlier project, Adam Harvey explored the use of tribal paint and high-fashion aesthetics to defeat automatic face detection systems.⁶ By contrast, the type of makeup we considered is not intended to deliberately undermine the security of a biometric system but rather to improve facial aesthetics using commonly available cosmetics (see Figures 3–5).

We used four face recognition algorithms, or matchers: Gabor wavelets,⁷ local binary pattern,⁸ the commercial Verilook Face Toolkit,⁹ and local Gabor binary pattern.¹⁰

We considered 99 subjects from the YouTube MakeUp tutorials dataset. There are four images associated with each subject: two with makeup, and two without. To assess the impact of makeup, we conducted three different matching experiments for each subject (see Figure 6). In the first, face images without makeup were matched (N vs N). In the second, one of the images to be matched had makeup, while the other did not (M vs N). In the third experiment, both images to be matched had makeup (M vs M).

Figure 6. Results of face matching using four face recognition algorithms on a YMU sample. ‘N’ is the subject without makeup and ‘M’ the subject with makeup. Lower scores represent a higher similarity. LBP: Local binary pattern. LGBP: Local Gabor binary pattern. (Images obtained from YouTube).

Figure 7. Boxplot for the distance scores when matching images of the same subject using a commercial face matcher. The boxplots correspond to the three matching scenarios: M vs M, M vs N, and N vs N. Note the increase in distance (hence decrease in similarity) for M vs N.

Figure 8. The receiver operating characteristic curves of the commercial face matcher. Note the decrease in performance, indicated by the higher equal error rate (EER) for M vs N.

Figure 7 shows the decrease in similarity when comparing face images of a subject before and after makeup. Figure 8 plots the false non-match rate (FNMR) against the false match rate (FMR) at various matching score thresholds. FNMR is representative of a non-match between two face images of the same subject, while FMR is representative of an incorrect match between face images of two different individuals. The general observation here is that the equal error rate (EER, where FNMR=FMR) for the case M vs N is substantially higher than both the M vs M and N vs N cases. The results show that the application of makeup can be a substantial challenge to automated face recognition.

Alterations by cosmetic facial makeup are predominantly color-based, but our experiments show that they also affect the performance of face matchers based on gray scale images. Our analysis indicated that the application of eye makeup has the most pronounced effect.⁵ Since makeup is in widespread use, these results indicate the possibility of compromising the security of a face or periocular biometric system.¹¹

We have designed a method to automatically detect the presence of makeup. Furthermore, we have developed an adaptive pre-processing scheme that exploits information about the presence or absence of makeup to improve the matching accuracy of face recognition.¹² Our future work will seek to establish a more detailed experimental protocol that quantifies the degree of makeup applied to a subject's face, and to develop algorithms that are robust to changes introduced by makeup. We considered only female subjects in this work, but are designing a more elaborate experiment involving males and females. We also plan to explore the possibility of makeup-induced face alteration for spoofing purposes. We will study the impact of makeup on automated gender recognition and age estimation schemes. Additionally, we will analyze its impact on automated beauty assessment methods, which determine the aesthetic attribute of a face image.

This work was funded by the Center for Identification Technology Research.