Modern military operations utilize complex technologies that require high levels of readiness and sustained cognitive and physical performance of combat military combat personnel. These military operations often depend on weapon systems that use advanced computer technology coupled with an array of sensors that provide continuous information on the battlefield environment and on equipment function. However there is a lack of real-time information on status of the personnel who control these systems and who are vital to mission success. Failure of the human element renders the weapon system useless so it is important to know if an individual is physically and cognitively fit to perform his or her task. Based on the premise that status of metabolic processes provide an early indication of a change in an individuals physiological status, monitoring of selective biomarkers of metabolism and organ function can provide insight on the individual’s ability to perform mission tasks. During combat individuals may not be aware that they have reached a compromised physiological condition due to dehydration, physical exertion, stress, fatigue, sleep deprivation, exposure to toxins or other condition that may affect physical and cognitive performance and health. Systems that can provide the individual or his or her commander with information about significant changes in one or more metabolic functions could permit timely intervention to correct the condition. In the event that serious injury has already occurred to an individual, metabolic monitoring can provide valuable intelligence needed for decisions on achieving mission objectives.

The objectives of two Air Force Small Business research topics were to develop a real-time, unobtrusive, biological sensing and monitoring technology for evaluating cognitive readiness in command and control environments (i.e., console operators). We sought an individualized status monitoring system for command and control operators and teams. The system was to consist of a collection of bio-sensing technologies and processing and feedback algorithms that could eventually guide the effective incorporation of fatigue-adaptive workload interventions into weapon systems to mitigate episodes of cognitive overload and lapses in operator attention that often result in missed signals and catastrophic failures. Contractors set about determining what electro-physiological and other indicators of compromised operator states are most amenable for unobtrusive monitoring of psychophysiological and warfighter performance data. They proposed multi-sensor platforms of bio-sensing technologies for development. The sensors will be continuously-wearable or off-body and will not require complicated or uncomfortable preparation. A general overview of the proposed approaches and of progress toward the objective is presented.

Until recently scientists had limited opportunities to study human cognitive performance in non-laboratory, fully ambulatory situations. Recently, advances in technology have made it possible to extend behavioral assessment to the field environment. One of the first devices to measure human behavior in the field was the wrist-worn actigraph. This device, now widely employed, can acquire minute-by-minute information on an individual’s level of motor activity. Actigraphs can, with reasonable accuracy, distinguish sleep from waking, the most critical and basic aspect of human behavior. However, rapid technologic advances have provided the opportunity to collect much more information from fully ambulatory humans. Our laboratory has developed a series of wrist-worn devices, which are not much larger then a watch, which can assess simple and choice reaction time, vigilance and memory. In addition, the devices can concurrently assess motor activity with much greater temporal resolution then the standard actigraph. Furthermore, they continuously monitor multiple environmental variables including temperature, humidity, sound and light. We have employed these monitors during training and simulated military operations to collect information that would typically be unavailable under such circumstances. In this paper we will describe various versions of the vigilance monitor and how each successive version extended the capabilities of the device. Samples of data from several studies are presented, included studies conducted in harsh field environments during simulated infantry assaults, a Marine Corps Officer training course and mechanized infantry (Stryker) operations. The monitors have been useful for documenting environmental conditions experienced by wearers, studying patterns of sleep and activity and examining the effects of nutritional manipulations on warfighter performance.

The Cognitive Drug Research (CDR) computerized assessment system has been in use in worldwide clinical trials for over 20 years. It is a computer based system which assesses core aspects of human cognitive function including attention, information, working memory and long-term memory. It has been extensively validated and can be performed by a wide range of clinical populations including patients with various types of dementia. It is currently in worldwide use in clinical trials to evaluate new medicines, as well as a variety of programs involving the effects of age, stressors illnesses and trauma upon human cognitive function. Besides being highly sensitive to drugs which will impair or improve function, its utility has been maintained over the last two decades by constantly increasing the number of platforms upon which it can operate. Besides notebook versions, the system can be used on a wrist worn device, PDA, via tht telephone and over the internet. It is the most widely used automated cognitive function assessment system in worldwide clinical research. It has dozens of parallel forms and requires little training to use or administer. The basic development of the system wil be identified, and the huge databases (normative, patient population, drug effects) which have been built up from hundreds of clinical trials will be described. The system is available for use in virtually any environment or type of trial.

A hybrid sustained attention task was developed in order to examine the relationships between manual response times and the timing and morphology of horizontal saccades involved in shifting gaze to a source of task relevant visual information. Twelve subjects performed this task for 60 min with no breaks. Performance and gaze control measures were aggregated across 20 min intervals comprising early, middle and late segments of the task. Response time variability was significantly increased during later task segments (p<0.05). These segments were also associated with increased variability in the amplitude of saccades (p<0.05). Saccade durations during the late task segments were also longer and more variable (p<0.05). Correlations between response times and measure of saccadic activity were also computed across consecutive 5 min intervals for each individual subject. The obtained correlations between saccade latency and response times exceeded 0.70 for six of the twelve subjects. Additional analyses examined the relationship between trials characterized by extreme values on either the performance or the gaze control measures. Trials characterized by extremely long response times were also associated with increased saccade amplitudes, durations and latencies (p<0.01). Conversely, response times were abnormally long on trials categorized as extreme on the basis of the saccade morphology and timing measures (p<0.01). These results confirm the utility of the sustained attention task as a laboratory platform for the development of real-time systems for alertness monitoring. The data also support the contention that measures of gaze control behavior can reflect aspects of cognitive activity and, therefore, should be seriously considered for inclusion in any physiologically-based alertness assessment battery.

Personnel are often required to perform multiple simultaneous tasks at the limits of their cognitive capacity. In research surrounding cognitive performance resources for tasks during stress and high cognitive workload, one area of deficiency is measurement of individual differences. To determine the amount of attentional demand a stressor places on a subject, one must first know that all subjects are performing at the same level with the same amount of available capacity in the control condition. By equating the baselines of performance across all subjects (“baselining”) we can control for differing amounts of performance capacity or attentional resources in each individual. For example, a given level of task performance without a time restriction may be equated across subjects to account for attentional resources. Training to a fixed level of proficiency with time limits might obliterate individual differences in mental resources. Eye movement parameters may serve as a real-time measure of attentional demand. In implementing a baselining technique to control for individual differences, eye movement behaviors will be associated with the true cognitive demands of task loading or other stressors. Using eye movement data as a proxy for attentional state, it may be possible to “close the loop” on the human-machine system, providing a means by which the system can adapt to the attentional state of the human operator. In our presentation, eye movement data will be shown with and without the benefit of the baselining technique. Experimental results will be discussed within the context of this theoretical framework.

Recent evidence indicates that poor autonomic regulation, indexed by decreased heart period variability (HPV), is associated with decreased working memory. HPV analyses are computed on the interbeat interval time series derived from the electrocardiogram (EKG). Unfortunately, the duration of the data collection and the issue of the size of ambulatory monitors with sufficient storage capacity for multi-day records is somewhat problematic. In the present paper we describe a system that allows for the collection of large amounts of high quality data using a small data collection device. The recording system consists of a miniature, single-module electrocardiogram-recording device. This module consists of an integrated three-electrode device that is attached to the chest of the subject. A low power 8-bit micro-controller detects the R-spike and stores the time between R-spikes in milliseconds on a 512 KB EEPROM. This system can record continuously for over four days. This system will allow the recording of cardio-dynamics in the field and provide highly reliable data across multiple days. The use of this device to assess physiological function in military operations would allow researchers to examine longer data records across several contexts and to understand the role of changes in autonomic function as they relate to performance.

Measuring heart rate variability is an important component of developing human monitoring systems for soldiers of the next century. Unfortunately, even the best sensors are prone to error in active situations. We have developed a system that detects and corrects errors in interbeat interval data in real time. A six to ten second buffer is used to provide context for a set of rules designed to simulate the way a human expert corrects data offline. Interbeat interval data was gathered from a pool of eighteen subjects with three detection devices used on each subject. Results of the automated correction were compared with human experts to determine the validity of the method. As expected, success varied based on the number of errors in a neighborhood. Isolated errors were corrected with high accuracy, while severely damaged data streams were totally unrecoverable by human or machine. This technique could serve as a crucial component of interbeat interval based monitoring technologies.

In the present paper we describe a model of neurovisceral integration in which a set of neural structures involved in cognitive, affective, and autonomic regulation are related to heart rate variability (HRV) and cognitive performance. We will provide pharmacological and neuroimaging data in support of the neural structures linking the central nervous system to HRV. Next, we will review a number of studies from our group using military cadets showing that individual differences in HRV are related to performance on tasks associated with executive function and prefrontal cortical activity. In the first study, individual differences in resting HRV we related to performance on executive and non-executive function tasks. The results showed that greater HRV was associated with better performance on executive function tasks. In the second study we add a stressor (shock avoidance) to the previous paradigm and show that those with greater HRV were more stress tolerant. Specifically, those with greater HRV were not adversely affected by the added stressor. In the last experiment, HRV was manipulated by physical detraining. Again, those that maintained their HRV at the post-test showed better performance on executive function tasks. We propose that these findings have important implications for the development of biomarkers related to performance in modern warfighters.

Electroencephalographic (EEG) and neurocognitive measures were simultaneously acquired to quantify alertness from 24 participants during 44-hours of sleep deprivation. Performance on a three-choice vigilance task (3C-VT), paired-associate learning/memory task (PAL) and modified Maintenance of Wakefulness Test (MWT), and sleep technician-observed drowsiness (eye-closures, head-nods, EEG slowing) were quantified. The B-Alert system automatically classifies each second of EEG on an alertness/drowsiness continuum. B-Alert classifications were significantly correlated with technician-observations, visually scored EEG and performance measures. B-Alert classifications during 3C-VT, and technician observations and performance during the 3C-VT and PAL evidenced progressively increasing drowsiness as a result of sleep deprivation with a stabilizing effect observed at the batteries occurring between 0600 and 1100 suggesting a possible circadian effect similar to those reported in previous sleep deprivation studies. Participants were given an opportunity to take a 40-minute nap approximately 24-hours into the sleep deprivation portion of the study (i.e., 7 PM on Saturday). The nap was followed by a transient period of increased alertness. Approximately 8 hours after the nap, behavioral and physiological measures of drowsiness returned to levels prior to the nap. Cluster analysis was used to stratify individuals into three groups based on their level of impairment as a result of sleep deprivation. The combination of B-Alert and neuro-behavioral measures may identify individuals whose performance is most susceptible to sleep deprivation. These objective measures could be applied in an operational setting to provide a “biobehavioral assay” to determine vulnerability to sleep deprivation.

The integration of real-time electroencephalogram (EEG) workload indices into the man-machine interface could greatly enhance performance of complex tasks, transforming traditionally passive human-system interaction (HSI) into an active exchange where physiological indicators adjust the interaction to suit a user’s engagement level. The envisioned outcome is a closed-loop system that utilizes EEG and other physiological indices for dynamic regulation and optimization of HSI in real-time. As a first step towards a closed-loop system, five individuals performed as identification supervisors (IDSs) in an Aegis command and control (C²) simulated environment, a combat system with advanced, automatic detect-and-track, multi-function phased array radar. The Aegis task involved monitoring multiple data sources (i.e., missile-tracks, alerts, queries, resources), detecting required actions, responding appropriately, and ensuring system status remains within desired parameters. During task operation, a preliminary workload measure calculated in real-time for each second of EEG and was used to manipulate the Aegis task demands. In post-hoc analysis, the use of a five-level workload measure to detect cognitively challenging events was evaluated. Events in decreasing order of difficulty were track selection-identification, alert-responses, hooking-tracks, and queries. High/extreme EEG-workload occurred during high cognitive-load tasks with a detection efficiency approaching 100% for selection-identification and alert-responses, 77% for hooking-tracks and 70% for queries. Over 95% of high/extreme EEG-workload across participants occurred during high-difficulty events (false positive rate < 5%). The high/extreme workload occurred between 25-30% of time. These results suggest an intelligent closed-loop system incorporating EEG-workload measures could be designed to re-allocate tasks and aid in efficiently streamlining a user’s cognitive workload. Such an approach could ensure the operator remains uninterrupted during high/extreme workload periods, thereby resulting in increased productivity and reduced errors.

Mission success in military operations depends upon optimal functioning of all system components, including the human operator. The cognitive demands of current systems can exceed the capabilities of the human operator. In some situations, such as Unmanned Combat Air Vehicle (UCAV) operations, one operator may be required to supervise several vehicles simultaneously. The functional state of the human operator is not currently considered in the overall system assessment. It has been assumed that the operator could “manage” any situation given a well designed system. However, with the requirement to monitor and remotely monitor several vehicles simultaneously during combat comes the possibility of cognitive overload. This increases the probability of committing errors. We have developed on-line measures of operator functional state using psychophysiological measures. These measures provide estimates of how well an operator can deal with the current task demands. When the operator is cognitively overloaded the system may be able to implement adaptive aiding procedures. This will reduce the task demands on the operator thereby improving mission success. We have demonstrated correct assessment of the functional state of the operator with accuracies of 95% or better. Psychophysiological measures were used with classifiers such as artificial neural networks. In one study, adaptive aiding was applied when the classifier determined operator overload. The aiding resulted in significantly improved performance.

We measured multichannel EEG spectra during a continuous mental arithmetic task and created statistical learning models of cognitive fatigue for single subjects. Sixteen subjects (4 F, 18-38 y) viewed 4-digit problems on a computer, solved the problems, and pressed keys to respond (inter-trial interval = 1 s). Subjects performed until either they felt exhausted or three hours had elapsed. Pre- and post-task measures of mood (Activation Deactivation Adjective Checklist, Visual Analogue Mood Scale) confirmed that fatigue increased and energy decreased over time. We examined response times (RT); amplitudes of ERP components N1, P2, and P300, readiness potentials; and power of frontal theta and parietal alpha rhythms for change as a function of time. Mean RT rose from 6.7 s to 7.9 s over time. After controlling for or rejecting sources of artifact such as EOG, EMG, motion, bad electrodes, and electrical interference, we found that frontal theta power rose by 29% and alpha power rose by 44% over the course of the task. We used 30-channel EEG frequency spectra to model the effects of time in single subjects using a kernel partial least squares (KPLS) classifier. We classified 13-s long EEG segments as being from the first or last 15 minutes of the task, using random sub-samples of each class. Test set accuracies ranged from 91% to 100% correct. We conclude that a KPLS classifier of multichannel spectral measures provides a highly accurate model of EEG-fatigue relationships and is suitable for on-line applications to neurological monitoring.

In one experiment, EEG recordings were made during a daytime session while 16 well-rested participants performed versions of a PC flight simulator task that were either low, moderate, or high in difficulty. In another experiment, the same subjects repeatedly performed high difficulty versions of the same task during an all night session with total sleep deprivation. Multivariate EEG metrics of cortical activation were derived for frontal brain regions essential for working memory and executive control processes that are presumably important for maintaining situational awareness, central brain regions essential for sensorimotor control, and posterior parietal and occipital regions essential for visuoperceptual processing. During the daytime session each of these regional measures displayed greater activation during the high difficulty task than during the low difficulty task, and degree of cortical activation was positively correlated with subjective workload ratings in these well-rested subjects. During the overnight session, cortical activation declined with time-on-task, and the degree of this decline over frontal regions was negatively correlated with subjective workload ratings. Since participants were already highly skilled in the task, such changes likely reflect fatigue-related diminishment of frontal executive capability rather than practice effects. These findings suggest that the success of efforts to gauge mental workload via proxy cortical activation measures in the context of adaptive automation systems will likely depend on use of user models that take both task demands and the operator’s state of alertness into account. Further methodological development of the measurement approach outlined here would be required to achieve a practical, effective objective means for monitoring transient changes in cognitive brain function during performance of complex real-world tasks.

Progress is described in developing a novel test of neurocognitive status for fitness-for-duty testing. The Sustained Attention & Memory (SAM) test combines neurophysiologic (EEG) measures of brain activation with performance measures during a psychometric test of sustained attention and working memory, and then gauges changes in neurocognitive status relative to an individual’s normative baseline. In studies of the effects of common psychoactive substances that can affect job performance, including sedating antihistamines, caffeine, alcohol, marijuana, and prescription medications, test sensitivity was greater for the combined neurophysiological and performance measures than for task performance measures by themselves. The neurocognitive effects of overnight sleep deprivation were quite evident, and such effects predicted subsequent performance impairment on a flight simulator task. Sensitivity to diurnal circadian variations was also demonstrated. With further refinement and independent validation, the SAM Test may prove useful for assessing readiness-to-perform in high-asset personnel working in demanding, high risk situations.

Submarine watch-standers adhere to a 6 hour-on, 12 hour-off (6/12) watch-standing schedule, yoking them to an 18-hr day, engendering circadian desynchronization and chronic sleep deprivation. Moreover, the chronic social crowding, shift work, and confinement of submarine life provide additional stressors known to correlate with elevated secretory immunoglobulin A (sIgA) and cortisol levels, reduced performance, immunologic dysfunction, malignancies, infections, gastrointestinal illness, coronary disease, anxiety, and depression. We evaluated an alternative, compressed, fixed work schedule designed to enhance circadian rhythm entrainment, sleep hygiene, performance, and health on 10 underway submariners, who followed the alternative and 6/12 schedules for approximately 2 weeks each. We measured subjects’ sleep, cognitive performance, and salivary biomarker levels. Pilot analysis of the salivary data on one subject utilizing ELISA suggests elevated biomarker levels of stress. Average PM cortisol levels were 0.2 μg/L (normal range: nondetectable - 0.15 μg/L), and mean sIgA levels were 562 μg/ml (normal range: 100-500 μg/ml). Future research exploiting real-time salivary bioassays, via fluorescent polarimetry technology, identified by the Office of Naval Research (ONR) as a future Naval requirement, allows researchers to address correlations between stress-induced elaboration of salivary biomarkers with physiological and performance decrements, thereby fostering insight into the underway submariner’s psychoimmunological status. This may help identify strategies that enhance resilience to stressors. Specifically, empirically-based modeling can identify optimal watch-standing schedules and stress-mitigating procedures -- within the operational constraints of the submarine milieu and the mission --that foster improved circadian entrainment and reduced stress reactivity, enhancing physiological health, operational performance, safety, and job satisfaction.

We explore the utility of functional Near Infra Red (fNIR) technology in providing both empirical support and a basis for assessing and predicting dynamic changes in cognitive workload within the theoretical context of computational cognitive modeling (CCM). CCM has had many successes and in recent years has expanded from a tool for basic research to one that can tackle more complex real-world tasks. As a tool for basic research it seeks to provide a model of cognitive functionality; as a tool for cognitive engineering it seeks applications in monitoring and predicting real-time performance. With this powerful theoretical tool we combine the empirical power of fNIR technology. The fNIR technology is used to non-invasively monitor regional hemodynamic activities, namely blood volume changes and oxygenation dynamics. We examined a simple auditory classification task in four different workload conditions. We monitored the blood activity in the prefrontal cortex region of the frontal lobe during the performance of the task to assess the patterns of activity as workload changes. We associated patterns of model activity with patterns of the hemodynamic data. We used ACT-R for creating the computational cognitive model. For the fNIR analysis, we used a generalized linear regression model along with time series clustering. We found that in the highest workload condition the model predicts a cognitive 'overload', which correlated well with the fNIR cluster and classification analysis, as this condition differs significantly from the other three conditions. Linear regression on a subset of the data where workload increases monotonically shows that apart from the overload condition, there was a positive relationship between increase in workload and increase in blood volume activation. In addition, individual variations in hemodynamic response suggest that individuals differ in how they process different workload levels.

Adverse effects of intoxication, fatigue and boredom could degrade performance of highly trained operators of complex technical systems with potentially catastrophic consequences. Existing physiological fitness for duty tests are time consuming, costly, invasive, and highly unpopular. Known non-physiological tests constitute a secondary task and interfere with the busy workload of the tested operator. Various attempts to assess the current status of the operator by processing of "normal operational data" often lead to excessive amount of computations, poorly justified metrics, and ambiguity of results. At the same time, speech analysis presents a natural, non-invasive approach based upon well-established efficient data processing. In addition, it supports both behavioral and physiological biometric. This paper presents an approach facilitating robust speech analysis/understanding process in spite of natural speech variability and background noise. Automatic speech recognition is suggested as a technique for the detection of changes in the psycho-physiological state of a human that typically manifest themselves by changes of characteristics of voice tract and semantic-syntactic connectivity of conversation. Preliminary tests have confirmed that the statistically significant correlation between the error rate of automatic speech recognition and the extent of alcohol intoxication does exist. In addition, the obtained data allowed exploring some interesting correlations and establishing some quantitative models. It is proposed to utilize this approach as a part of fitness for duty test and compare its efficiency with analyses of iris, face geometry, thermography and other popular non-invasive biometric techniques.

Sending astronauts into space, especially on long-durations missions (e.g. three-year missions to Mars), entails enormous risk. Threats include both physical dangers of radiation, bone loss and other consequences of weightlessness, and also those arising from interpersonal problems associated with extended life in a high-risk isolated and confined environment. Before undertaking long-duration missions, NASA seeks to develop technologies to monitor indicators of potentially debilitating stress at both the individual and team level so that countermeasures can be introduced to prevent further deterioration. Doing so requires a better understanding of indicators of team health and performance. To that end, a study of team problem solving in a simulation environment was undertaken to explore effects of team and task stress. Groups of four males (25-45 yrs) engaged in six dynamic computer-based Antarctic search and rescue missions over four days. Both task and team stressors were manipulated. Physiological responses (ECG, respiration rate and amplitude, SCL, EMG, and PPG); communication (voice and email); individual personality and subjective team dynamics responses were collected and related to task performance. Initial analyses found that physiological measures can be used to identify transient stress, predict performance, and reflect subjective workload. Muscle tension and respiration were the most robust predictors. Not only the level of arousal but its variability during engagement in the task is important to consider. In general, less variability was found to be associated with higher levels of performance. Individuals scoring high on specific personality characteristics responded differently to task stress.

This paper explores how the accuracy of a first-principles physiological model can be enhanced by integrating data-driven, "black-box" models with the original model to form a "hybrid" model system. Both linear (autoregressive) and nonlinear (neural network) data-driven techniques are separately combined with a first-principles model to predict human body core temperature. Rectal core temperature data from nine volunteers, subject to four 30/10-minute cycles of moderate exercise/rest regimen in both CONTROL and HUMID environmental conditions, are used to develop and test the approach. The results show significant improvements in prediction accuracy, with average improvements of up to 30% for prediction horizons of 20 minutes. The models developed from one subject's data are also used in the prediction of another subject's core temperature. Initial results for this approach for a 20-minute horizon show no significant improvement over the first-principles model by itself.

Motion sickness (MS) is a stressor commonly attributed with causing serious navigational and performance errors. The distinct nature of MS suggests this state may have distinct neural markers distinguishable from other states known to affect performance (e.g., stress, fatigue, sleep deprivation, high workload). This pilot study used new high-resolution electro-encephalograph (EEG) technologies to identify distinct neuronal activation changes that occur during MS. Brain EEG activity was monitored while subjects performed a ball-tracking task and viewed stimuli on a projection screen intended to induce motion sickness/spatial disorientation. Results show the presence of EEG spectral changes in all subjects who developed motion sickness when compared to baseline levels. These changes included: 1) low frequency (1 to 10 Hz) changes that may reflect oculomotor movements rather than intra-cerebral sources; 2) increased spectral power across all frequencies (attributable to increased scalp conductivity related to sweating), 3) local increases of power spectra in the 20-50 Hz range (likely attributable to external muscles on the skull) and; 4) a central posterior (occipital) independent component that shows suppression of a 20 Hz peak in the MS condition when compared to baseline. Further research is necessary to refine neural markers, characterize their origin and physiology, to distinguish between motion sickness and other states and to enable markers to be used for operator state monitoring and the designing of interventions for motion sickness.