A novel convolutional neural network for deep-learning classification

Brain–computer interfaces (BCIs) have traditionally been used to enable communication and control for paralyzed patients.¹ However, it is also thought that BCIs hold promise for fulfilling the longstanding goal of creating artificial systems (i.e., which can perform with the adaptability, robustness, and general intelligence of humans). To augment the sensing and processing capabilities of such artificial systems, BCI systems can thus be used on healthy individuals. In this way, the biological machinery that enables human cognition can be leveraged. Image triage—a visual target search over a set of images—is a prime application for this new class of BCI. Humans can effortlessly identify target objects in scenes that stymie even the best machine vision techniques. Manual inspection by humans, however, is limited by the speed at which targets can be consciously detected and reported by a behavioral response. For example, when targets are identified by pressing a button, the button is typically pressed two to five images after the target image is shown (when the image stream is presented at 5Hz). This forces the observer to assume a distribution of images for the several images that precede the button press.² In addition, humans perform inconsistently because of exogenous distractions or endogenous factors (e.g., fatigue), whereas computer vision algorithms offer constant and predictable performance.

As an alternative, machine learning approaches can be applied to raw human neurophysiological data and thus reveal signals that are relevant to the detection of target images. Ultimately, this can increase both the accuracy and the response rate of image triage classification tasks. Indeed, in recent work, it has been shown that the classification performance in a rapid serial visual presentation (RSVP) image triage task (see Figure 1) can be improved by combining human neurophysiological data with machine vision classifiers.² To date, such methods have relied on the late fusion of human and machine-generated classifier outputs. In other words, the classifiers for image and human data are trained separately and their outputs are later fused. It may be possible to improve the classification performance even further if the complementary information carried in the human signals and the image data can be trained in tandem. To realize this aim, however, relevant neurophysiological data (which carries a discriminatory signal) and the ability to process and convert these signals to useful task determinants is required. In addition, it is necessary to have a common framework, within which it is possible to train a classifier that directly learns combined models of human neurophysiological data and image data.

Figure 1. Illustrating the display of images during a rapid serial visual presentation (RSVP) task. In this case, the images are presented at 5Hz and the subjects are required to indicate (via a behavioral response) when an—infrequently occurring—target image is shown.

The aim of our work³ is to improve human-autonomy classification performance by developing a single framework that builds codependent models of human neuophysiological information and image data to generate fused target estimates. CNNs are a type of supervised deep-learning architecture that have set record benchmarks in many domains, including speech recognition, drug discovery, genomics, and visual object recognition.⁴ CNNs enable automatic feature selection and extraction from raw data. This is achieved by hierarchically stacking linear and nonlinear filtering modules to form a network, where each layer transforms an input into a representation at a higher and more abstract level. The resulting non-convex optimization is performed through the iterative application of a back-propagation algorithm until the maximum performance is achieved or the network converges. Given the success of using CNNs in machine vision for object recognition, as well as recent work in which CNNs are used for multimodal fusion,^5–7 we believe that CNNs are thus promising for early fusion models of EEG data and computer vision.

As a first step in our study, we investigated the use of CNNs for multiclass single-trial classification of EEG recordings across multiple subjects during an RSVP task. Our results suggest that the EEG RSVP CNN classifier was able to meet—and exceed—the performance of other classifiers. This was the case for a single generalized model across 18 subjects (where our method learned one model and each of the other classifiers learned 18 models). Our classifier also out-performed the other classifiers for automatically selecting features that do not explicitly rely on the detection of any known features. A sample of the filters learned by our network in these tests is shown in Figure 2.

Figure 2. Three sample layers of spatial filters learned by the electroencephalograph (EEG) RSVP convolution neural network. The filters are shown mapped to the position of EEG electrodes that were placed on the scalp of subjects. Dark red and dark blue indicate larger magnitudes of activation (no units). Black dots denote the true spatial location of each electrode on the scalp.

Our CNN design includes four convolutional layers, two fully connected layers, and a readout layer. We provide a more detailed rationale for this network architecture, and its hyperparameters, elsewhere.³ Our preliminary results (see Figure 3) show that—compared with the other classifiers—our CNN achieves the highest performance level. The area-under-the-curve (AUC) value we obtain (0.72) is higher than for the second-best classifier (0.71), despite the fact that we use a generalized model and individual models are used for the other classifier. In addition, when we train our classifier to convergence, we see overfitting and a corresponding increase in loss. To prevent this, we only train the classifier until the loss on the validation set starts to increase.

Figure 3. Area under the curve (AUC) and loss of the EEG RSVP CNN classifier during training iteration on the evaluation testing set. The dashed red line indicates when the maximum AUC value was reached.

In summary, we have designed a CNN deep-learning classifier that learns a single generalized model across multiple subjects for single-trial RSVP EEG classification. We have demonstrated that our CNN is a viable alternative to existing neural classifiers, by showing that it meets and exceeds the classification performance of several leading classifiers. By designing a CNN classifier to automatically detect the features that maximally separate target and non-target samples from raw data, we hypothesized that our framework would learn a feature set that enabled high performance. We also predicted that the classifier would have an increased robustness across subjects. Our preliminary analysis, however, is not sufficient to make meaningful statistical comparisons between the performance of our network and the state-of-the-art methods. Nonetheless, we are able to achieve a similar performance with our CNN (i.e., without requiring individualized models for each subject or explicit reliance on known features in the raw data). We are currently extending this work to statistically validate the performance of our CNN. We will quantify the contributions of different spatial weights and analyze the significance of the features that the network has learned.