Brain-computer interface games for hedonic experiences

Conventional computer interfaces, such as a keyboard or a mouse, are limited in the information they provide. For instance, they cannot tell computers about actions we intend to perform or imagine performing (e.g., moving a limb) or mental states we are in (e.g., emotional or attentional state). In contrast, a brain-computer interface (BCI) can provide this information. BCIs interpret our brain activity by analyzing our brain signals. These are mostly electrical signals measured by electrodes placed in contact with the scalp (electroencephalography, EEG).

Using a BCI implies mutual learning between us and the computer. The computer needs to adapt to our brain's signal characteristics while we need to learn how to generate the brain signals that can be recognised by the computer. Learning might not be desirable for pragmatic applications used for efficient control, but it might be preferred in hedonic applications used for fun. For example, one motivation in playing games is to experience virtuosity, that is, when we learn to overcome a difficulty. So, while playing a BCI game, we might consider the learning process as a challenge and experience virtuosity as we gain control in the game.^{1, 2}

To investigate the experience that the challenge of playing a BCI game yields, we developed a multi-modal game called Mind the Sheep! that is played partly using a mouse and partly using BCI.^{3, 4} The game world contains fences, a number of sheep that move autonomously, three dogs that are moved by players, and other elements representing a meadow: see Figure 1). The aim is to fence the sheep in as quickly as possible by herding them with their dogs. Players select a dog and move it around the game world. To select a dog, the players keep the left mouse button pressed. As long as the button is pressed, the dog images are replaced by circles that flicker (i.e., alternate between black and white colour) at distinct frequencies (e.g. 7.5, 10, and 12Hz): see Figure 2). The players focus their vision on the flickering circle that replaces the dog they wish to select. This produces a brain response called steady-state visually evoked potential (SSVEP) which enhances the EEG signal amplitude at the same frequency as the player's selected flickering circle.⁵ The more the player pays attention, the stronger the SSVEP response. When the mouse button is released, the EEG signals are subjected to frequency analysis,⁶ one of the dogs is selected, and the selected dog moves to the location of the cursor. Our pilot experiments showed that, on average, 84% of the time the game correctly identified the player's selected dog. This SSVEP-based attention recognition was significantly above the level of chance but still imperfect, providing players with the challenge of focusing better and gaining more control.

Figure 1. Screenshot from the game Mind the Sheep!

Figure 2. Screenshot while selecting a dog in Mind the Sheep!

Figure 3. Two people discussing while playing Mind the Sheep!

The game can be played by a single player as well as by multiple players collaboratively: see Figure 3). There are several other versions of the game in which the dogs are selected by pronouncing their names or by clicking on the dog images. This variety of selection methods permits comparative user experience studies.

To investigate the influence of the challenge offered by Mind the Sheep! on user experience, we conducted three studies, experimenting with a total of 57 people.³ Participants played the game alone or with other people and we collected data about their playing experience using interviews, questionnaires, audio-visual recordings, and data logging. We investigated different indicators of user experience, such as immersion, affect, engagement, workload, and social interaction.

Our experiments showed that the challenge of playing a BCI game enhanced emotional social interaction between co-players but hindered collaborative social interaction due to a split-attention effect. Players felt that they were immersed in the game as they were cognitively involved in playing and became disassociated from the real world. The novelty of playing a BCI game strongly motivated them to play, and they were tolerant of errors. This also underlined the importance of considering long-term user experience, that is, how user experience would change as the novelty wears off.

In summary, we have demonstrated that it is possible to develop a challenging BCI game that provides players with a hedonic experience. Next, we would like to investigate how else—other than through challenge—BCI games can provide hedonic experiences. Besides SSVEP-based attention recognition, which we used in Mind the Sheep!, there are several other capabilities of BCIs that can be used in games. These include recognising workload, emotions, imaginary and preparatory movements, and responses to errors.⁷ We are particularly interested in exploiting BCI's capacity to recognise emotional states. One of the possibilities is to enhance people's presence in virtual environments (such as the game world) by representing their emotional states as well as their actions. Another direction is to foster affective bonds between people through BCI games that display and allow exchange of emotional states.

The authors gratefully acknowledge the support of the BrainGain Smart Mix Programme of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science.