Mid-wave IR face recognition systems

Today's military threats are different from those of the recent past. The organizations and skills necessary to develop efficient operations against such threats are not the same as even 10 years ago. The present-day army requires diverse capabilities to operate in the modern combat environment, and one of the high-demand skills of soldiers is the ability to detect and track humans as well as identify them. Face-based recognition (FR) is popular with the military for establishing human identity, because it has several advantages over other biometric traits: it is non-intrusive, understandable, and a facial image can be captured in a covert manner at variable standoff distances. Although varying factors such as illumination, cosmetics, and facial disguise can hinder FR performance, one of the biggest challenges is the ability to recognize a person in both day- and night-time environments. To mitigate such a challenge, FR operation in the IR spectrum (active and passive) has become increasingly important.

The IR spectrum is comprised of the active IR band (near-IR or short-wave IR, SWIR), and the thermal (passive) IR band. The passive IR band is further divided into the mid-wave (MWIR) and long-wave IR (LWIR) bands. The MWIR range is 3–5μm, whereas the LWIR range is 7–14μm. Both MWIR and LWIR cameras can sense temperature variations across human faces at a distance and produce thermograms in the form of 2D images. However, although both pertain to the thermal spectrum, they reveal different image characteristics of the facial skin (note that, in fact, thermal cameras are categorized as MWIR or LWIR). The difference between MWIR and LWIR is that MWIR has both reflective and emissive properties, whereas LWIR consists primarily of emitted radiation. There are three main advantages of MWIR over the active IR band. First, MWIR imagery can be acquired without any external illumination in day or night environments, while regions in the active IR band might require an external light source. Second, vein patterns (or other anatomical features) not observable in the active IR spectrum can be observable in MWIR. Finally, background clutter in MWIR images is not always visible. For example, the texture of a wall will not usually be visible if it is uniform and has the same surface temperature signature. Thus, when operating in the MWIR band, the tasks of face detection, localization, and segmentation—fundamental processes of typical face recognition systems—are comparatively easier and more reliable than in active IR and visible bands.

When data is acquired in the MWIR band, the camera sensor detects IR radiation in the form of heat that is emitted from subjects' faces. MWIR sensors are passive and make it feasible to acquire human biometric information even under obscure environments (e.g., at night) in a covert manner. Such a capability combined with other IR sensors (such as SWIR) can improve the performance of FR systems in the dark.

Figure 1. Sample face images in the (a) visible and (b) mid-wave IR (MWIR) bands after geometric normalization is applied. (c) MWIR-based features extracted using our proposed method.¹

Figure 2. Methodology used to perform fully automated pre-processing of MWIR-based image, including feature extraction, reference annotation, and matching.¹ SIFT: Scale-invariant feature transform.² SURF: Speeded up robust features.³

We have developed novel MWIR-based FR techniques that can be effective 24 hours a day in the field. The first technique enables the efficient detection of human eyes by combining a set of methodological steps such as face normalization, integral projections, and template-based matching (see Figure 1). The second technique permits the successful matching of acquired MWIR face images (also commonly known as probes) against a gallery dataset of MWIR face images (normally captured in a controlled environment of a set of individuals enrolled in the dataset). The method (see Figure 2) is composed of a pre-processing step (a fully automated standardization of MWIR images), a feature extraction step, and a matching step. After pre-processing, we use a statistically-based algorithm for physiological feature extraction (namely wrinkles, veins, edges, and perimeters of facial characteristics). Fiducial (that is, reference) points are subsequently detected either manually or automatically. For that purpose, we use three different extractors, i.e., a fingerprint-based minutiae detector, the scale-invariant feature transform (SIFT), and the speeded up robust features (SURF) detector.^{2, 3} Finally, faces are matched using an alignment-based matching algorithm with the ability of finding the correspondences between a stored set of gallery points and an input set of probe points.⁴

The MWIR camera used is capable of generating high-definition images and operating in diverse testing environments, featuring an indium antimonide focal plane array that achieves mega-pixel image resolution (1024×1024) per image.^{4, 5} We used the MWIR camera and a visible camera to collect facial images in both controlled indoor and uncontrolled outdoor environments. The standoff distance was set to 6.5 feet. In total, 50 subjects participated in our experiments, and the database has 15 indoor (controlled room-temperature environment) and 15 outdoor full-frontal thermal and visible face images of each subject, resulting in a total of 2250 images.⁶

We tested our first FR technique by performing a set of eye localization experiments, which showed that human eyes on frontal still MWIR face images can be detected with promising results. In particular, our template-based eye detection technique achieved the best accuracy when ocular templates (that is, templates of the eye and eyebrow regions) were used.⁶ In the main FR experiments, we first established a baseline by comparing a probe dataset of visible face images (queries) against a gallery dataset of visible images. We then compared an indoor dataset of MWIR face images against an either indoor or outdoor dataset. Finally, we studied the heterogeneous problem of matching MWIR (queries) against visible (gallery) face images (and vice versa).⁷

We evaluated FR performance on holistic face images and compared this against commercial and academic⁸ FR software. Experiments showed that images captured in the MWIR band can be efficiently matched to MWIR images using FR techniques that were not originally designed to address such a problem. However, the best results were obtained using our proposed FR approach.¹ We also found out that identification performance on MWIR imagery appears to be comparable to that of visible imagery (used as the baseline).

In the challenging FR experiments we noticed that when matching indoor (gallery dataset) to outdoor (probe dataset) MWIR images, we have a ∼25% performance drop across all FR algorithms used. This is mainly because of outdoor environmental effects on the appearance of the MWIR faces. As expected, cross-spectral experiments resulted in markedly reduced performance, before and after score level fusion of different methods.^{6, 7} In general, it appears that MWIR modality holds great promise under reasonable operating conditions. Another benefit of using MWIR is the possibility of obtaining the same recognition rates when face images are acquired in complete darkness.

Next, we intend to expand our collection and analysis effort to improve the performance of MWIR-based eye detection and face matching approaches. We will collect additional datasets and conduct further experiments to investigate more challenging scenarios, such as when subjects are wearing glasses, or when images are acquired only outdoors.

This work has been sponsored in part through the National Science Foundation Center for Identification Technology Research and by TechConnect, WV. The author gives special thanks to Nnamdi Osia, a doctoral student in the Multispectral Imagery Lab, as well as other active and former members of the biometrics team at West Virginia University, including Lawrence A. Hornak, Bojan Cukic, Arun Ross, and Zain Jafri.