Helmet-mounted displays to improve pilot situation awareness

In the past few decades, technological advances have significantly improved the design and engineering of aircraft. These developments have contributed to make pilot systems more complex and, hence, have increased the demand on the human operator. To add to this, helicopter pilots often fly in degraded visual environments. Poor visibility due to adverse weather or stirred up dust or sand during landing or take-off (brownout) can induce spatial disorientation, high workload, and loss of situation awareness. These factors pose a major problem not only for civil aviation pilots but especially in the military. Given the relatively high incident and accident rates in this field,¹ there is a need for enhanced visual support, which could be achieved by superimposing information on a helmet-mounted display (HMD).

When considering HMD implementation, it is crucial to take into account numerous human-factor issues, including attentional, perceptual, visual, and somatic aspects. Relating to attentional issues, aviation psychologists often discuss HMD assets and drawbacks in the framework of space- and object-based theories of attention. The former hypothesize that visual attention is, in principle, comparable to a spotlight: once the focus is on a given location, all information positioned within that location can be processed at the same time (parallel processing). This means that positioning elements relevant for a specific task in close spatial proximity enhances task performance. With regard to HMDs, this is one of the basic ideas behind superimposing instrument information on the pilot's forward field of view.

Figure 1. Different types of power line mast designs for helmet-mounted-display presentation.²

Object-based theories, on the other hand, hypothesize that attention does not depend on spatial proximity. Rather, it is directed to the objects' specific characteristics and grouping factors such as common color, texture, or movement. According to these theories, objects that share the same properties—including spatial position, size, and movement—capture attention. In the case of HMD symbology, this means the pilot would benefit from seeing information displayed conformally.³ In conformal symbology, the representation of the object (symbol) is displayed on the HMD in the same line of sight as the actual obstacle in the outside scene, and moving in accordance with the position of the pilot's head and the orientation of the aircraft. Previous studies indicate that conformal symbology accounts for divided attention tasks, i.e., tasks that require the integration of information from different sources, such as from the HMD display (near domain) and the outside scene (far domain).^{3, 4} It has also been shown to reduce attentional tunneling.⁵

In our work, we are investigating options to develop conformal HMD symbology based on two different approaches: using information obtained from a database or derived from sensors mounted on the aircraft.

Figure 2. Obstacle designs derived from sensor data fusion:⁶ ‘wire-frame’ (left), ‘opaque’ (center), and ‘thorned’ (right).

Within the database approach, we developed different obstacle designs (see Figure 1) and evaluated them in an online survey,² which also included a biographical questionnaire. A sample of 62 helicopter pilots provided ratings and suggestions for improvement. The biographical data obtained in the survey revealed that 59% of pilots had experienced severe spatial disorientation at least once and 75% reported having encountered at least one brownout during flight. Given the rather high occurrence of spatial disorientation, and the fact that a NATO study revealed that brownout is responsible for about 75% of helicopter mishaps,¹ the results clearly endorse the importance of augmented visual support. In addition, we found the cost of clutter to be the most crucial concern for pilots. They dislike symbologies with complicated designs because they contribute to more cluttered displays, which can impair performance by obscuring valuable information in the outside scene, among other drawbacks.

Within the sensor-based approach, we tested different obstacle presentations (see Figure 2)⁶ in a part-task simulation⁷ concentrating on aspects of focused, selective, and divided attention. We presented obstacles detected by the sensors mounted on the aircraft on the HMD as ground-based columns on a terrain grid with three types of design. In the ‘wire-frame’ design only the contours of those columns are highlighted, while in the ‘opaque’ display the entire columns are shaded. In the ‘thorned’ display the tops of the columns are peaked to represent potential danger.

We were interested in finding out whether these types of obstacle presentations could enhance situation awareness by facilitating the identification of landmarks. In addition, we wanted to know how compelling (in terms of attentional capture) the presentations were, and how this could affect divided and selective attention in the subjects. In a landmark-detection (focused-attention) task, pilots correctly identified 96% of the objects when wire-frame and opaque designs were used, and only 88% when the display used thorned symbology. A secondary task designed to test divided and selective attention required the pilots to respond to questions about current flight parameters. We evaluated correct responses, errors, and misses and concluded that pilots performed better with opaque obstacle design and worst with wire-frame symbology. We also verified that the pilots were more likely to answer a question correctly if it appeared on the HMD rather than on a head-down display. Our results indicate that a wire-frame obstacle design induces the highest clutter and tunnels the pilots' attention the most. Although this did not impair performance of focused-attention tasks on the HMD, it seemed to hinder performance of divided- and selective-attention tasks when directing attention outward. Overall, the opaque display did the best job, both in the primary and secondary tasks, as well as in the pilot ratings.

In summary, we investigated two alternative approaches to presenting obstacle information on an HMD to help helicopter pilots in poor visibility. To better understand which of the two methods is better for presenting information, we will next implement and test them in extensive simulator and flight trials. We will also examine ways of using conformal symbology to present routes, restricted areas, and landing zones, among others, in a pilot's HMD. Our test trial will contain a comprehensive analysis on human-factor aspects, including psychophysiological measurements, and will further examine effects on flight performance such as tracking performance and event detection.