Makeup challenges automated face recognition systems

Everyday makeup may reduce the accuracy of systems that use biometric matching for security screening.
15 April 2013
Antitza Dantcheva, Arun Ross and Cunjian Chen

The accuracy of face recognition systems has rapidly increased over the past decade, primarily due to robust feature representation and matching techniques. These methods are resilient to variations in facial pose, expression, and illumination.1 However, there are additional factors that compromise the use of these systems in security applications. These include aging—a natural biological change—and plastic surgery2—a medically induced change—both known to reduce accuracy (see Figure 1).

Figure 1. Challenges in face recognition: images of actress Jennifer Grey demonstrating variations due to pose, illumination, expression, and alterations through plastic surgery, aging and makeup. (Images obtained from,, and YouTube).

Another factor is non-permanent makeup that can alter the perceived shape and texture of a face. Makeup can be used strategically for spoofing or obfuscation (see Figure 2). In spoofing, an adversary modifies his facial appearance to resemble another individual; in obfuscation, he makes alterations to evade recognition by a biometric system. We have considered the possibility of unintentional obfuscation through the use of facial makeup.

Figure 2. The impact of makeup. (a) The subject before makeup application. (b-f) After application. (Images obtained from YouTube).

Makeup is a cost-efficient and socially acceptable way to substantially change the appearance of the face. Specifically, it can alter the apparent shape by accentuating contouring techniques (see Figure 3a). It can change the appearance of skin quality and color (see Figure 3b), alter the perceived nose shape and size, enhance or reduce the size of the mouth (see Figure 3c), and modify contrast by adding color. Makeup may alter the apparent form, shade, and location of the eyebrows (see Figure 5a, b), and the perceived shape, size, and contrast of the eyes (see Figure 4e–h). Other effects include concealing dark circles under the eyes (see Figure 5c, d). In addition, makeup can camouflage wrinkles, birth moles, scars, and tattoos. A vast cosmetics market attempts to improve facial aesthetics while projecting good health.

Figure 3. Makeup effects that significantly alter facial appearance. (a) Contouring. (b) Concealing. (c) Enhancing. (Images obtained from YouTube).

Figure 4. Four samples from the YouTube MakeUp (YMU) tutorials dataset.5Each subject is shown before applying makeup (a, c, e, g) and after (b, d, f, h). Two subjects have subtle makeup (b, d), and two heavy makeup (f, h). (Images obtained from YouTube).

Figure 5. How makeup alters the appearance of the eyes and eyebrows. (Images obtained from YouTube).

The impact of makeup on human ability to recognize faces has been studied by Ueda and Koyama.3 These authors observed that the application of light makeup slightly increases human recognizability, whereas heavy makeup significantly decreases it. Light makeup accentuates distinctive facial characteristics, thereby facilitating face recognition, while heavy makeup increases the bilateral size and symmetry of the eyes and lips, leading to a decreased characteristic distinctiveness.4 Light makeup is not easily perceived, since the applied colors are natural skin, lip, and eye colors (see Figure 4b, d). Heavy makeup is clearly visible, for example, red or dark lips, and strongly accentuated eyes (see Figure 4f, h). The notion of light or heavy makeup does not necessarily relate to the number of products used but rather to the difference in facial appearance before and after application (see Figure 3b).

We studied the impact of cosmetic makeup on automated face recognition systems. In an earlier project, Adam Harvey explored the use of tribal paint and high-fashion aesthetics to defeat automatic face detection systems.6 By contrast, the type of makeup we considered is not intended to deliberately undermine the security of a biometric system but rather to improve facial aesthetics using commonly available cosmetics (see Figures 35).

We used four face recognition algorithms, or matchers: Gabor wavelets,7 local binary pattern,8 the commercial Verilook Face Toolkit,9 and local Gabor binary pattern.10

We considered 99 subjects from the YouTube MakeUp tutorials dataset. There are four images associated with each subject: two with makeup, and two without. To assess the impact of makeup, we conducted three different matching experiments for each subject (see Figure 6). In the first, face images without makeup were matched (N vs N). In the second, one of the images to be matched had makeup, while the other did not (M vs N). In the third experiment, both images to be matched had makeup (M vs M).

Figure 6. Results of face matching using four face recognition algorithms on a YMU sample. ‘N’ is the subject without makeup and ‘M’ the subject with makeup. Lower scores represent a higher similarity. LBP: Local binary pattern. LGBP: Local Gabor binary pattern. (Images obtained from YouTube).

Figure 7. Boxplot for the distance scores when matching images of the same subject using a commercial face matcher. The boxplots correspond to the three matching scenarios: M vs M, M vs N, and N vs N. Note the increase in distance (hence decrease in similarity) for M vs N.

Figure 8. The receiver operating characteristic curves of the commercial face matcher. Note the decrease in performance, indicated by the higher equal error rate (EER) for M vs N.

Figure 7 shows the decrease in similarity when comparing face images of a subject before and after makeup. Figure 8 plots the false non-match rate (FNMR) against the false match rate (FMR) at various matching score thresholds. FNMR is representative of a non-match between two face images of the same subject, while FMR is representative of an incorrect match between face images of two different individuals. The general observation here is that the equal error rate (EER, where FNMR=FMR) for the case M vs N is substantially higher than both the M vs M and N vs N cases. The results show that the application of makeup can be a substantial challenge to automated face recognition.

Alterations by cosmetic facial makeup are predominantly color-based, but our experiments show that they also affect the performance of face matchers based on gray scale images. Our analysis indicated that the application of eye makeup has the most pronounced effect.5 Since makeup is in widespread use, these results indicate the possibility of compromising the security of a face or periocular biometric system.11

We have designed a method to automatically detect the presence of makeup. Furthermore, we have developed an adaptive pre-processing scheme that exploits information about the presence or absence of makeup to improve the matching accuracy of face recognition.12 Our future work will seek to establish a more detailed experimental protocol that quantifies the degree of makeup applied to a subject's face, and to develop algorithms that are robust to changes introduced by makeup. We considered only female subjects in this work, but are designing a more elaborate experiment involving males and females. We also plan to explore the possibility of makeup-induced face alteration for spoofing purposes. We will study the impact of makeup on automated gender recognition and age estimation schemes. Additionally, we will analyze its impact on automated beauty assessment methods, which determine the aesthetic attribute of a face image.

This work was funded by the Center for Identification Technology Research.

Antitza Dantcheva, Arun Ross
Michigan State University
East Lansing, MI

Antitza Dantcheva is a postdoctoral fellow in the Department of Computer Science and Engineering. She received her PhD in Signal and Image Processing in 2011 from Eurecom/Telecom ParisTech in France.

Arun Ross is an associate professor in the Department of Computer Science and Engineering. He is co-author of the books Introduction to Biometrics and Handbook of Multibiometrics.

Cunjian Chen
West Virginia University
Morgantown, WV

Cunjian Chen is a PhD candidate working with Arun Ross in the Lane Department of Computer Science and Electrical Engineering.

1. S. Z. Li, A. K. Jain, Handbook of Face Recognition, New York: Springer, 2011.
2. R. Singh, M. Vatsa, H. S. Bhatt, S. Bharadwaj, A. Noore, S. S. Nooreyezdan, Plastic surgery: a new dimension to face recognition, IEEE Trans. Inf. Forens. Secur. 5(3), p. 441-448, 2010.
3. S. Ueda, T. Koyama, Influence of make-up on facial recognition, Perception, 2010.
4. G. Rhodes, A. Sumich, G. Byatt, Are average facial configurations only attractive because of their symmetry?, Psychol. Sci. 10, p. 52-58, 1999.
5. A. Dantcheva, C. Chen, A. Ross, Can facial cosmetics affect the matching accuracy of face recognition systems?, Proc. 5th Int'l Conf. Biometrics: Theory Appl. Syst., 2012.
6. Homepage of the CV Dazzle (Camouflage from Computer Vision) project.
7. C. Liu, H. Wechsler, Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition, IEEE Trans. Image Process. 11(4), p. 467-476, 2002.
8. X. Tan, B. Triggs, Enhanced local texture feature sets for face recognition under difficult lighting conditions, IEEE Trans. Image Process. 19(6), p. 1635-1650, 2010.
9. Online information about the commercial Verilook Face Toolkit for face recognition.
10. W. Zhang, S. Shan, W. Gao, X. Chen, H. Zhang, Local Gabor binary pattern histogram sequence (LGBPHS): a novel non-statistical model for face representation and recognition, Proc. ICCV , p. 786-791, 2005.
11. U. Park, R. Jillela, A. Ross, A. K. Jain, Periocular biometrics in the visible spectrum, IEEE Trans. Inf. Forens. Secur. 6(1), p. 96-106, 2011.
12. C. Chen, A. Dantcheva, A. Ross, Automatic facial makeup detection with application in face recognition, Proc. ICB , 2013.
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?