Concept, development and evaluation of a mind action game with the electro encephalograms as an auxiliary input

—Games are interactive applications that require input devices in order to send messages for the interaction. Normally this input devices are mouse, keyboards and joy-sticks. Lately, this input has been done in different ways, such as voice, touch and movement with new input devices. One type of input that has not been very explored is the use of the brain waves as a input for the game. While in past these devices where expensive, nowadays Brain Computer Interface (BCI) have become accessible, cheap and can be acquired with nonintrusive top off-the-shelf products, which can create a new paradigm of interaction for games. This work presents a novel architecture and framework that can help the development of games with both BCI and traditional interfaces. As a proof of concept, this paper shows the experience in designing and developing a game prototype using the framework and EEG brainwaves as one of the players input. The game is an action slice game, similar to Fruit Ninja, called MindNinja. This game differ form most BCI game, since it is based on an action game, using touch input where the BCI is used as an auxiliary input to change the game behavior. This game was tested and evaluated with a group of person, showing promising results in the fun level, as well as increasing the attention level of subjects.


I. INTRODUCTION
The players immersion is one of the most important factors enhancing users experience and engagement in gaming.Immersion can be augmented by different methods: near reality graphics; realistic physical effects; believable artificial intelligence of non-player characters; or the use of a more natural input (like Kinect, Wii Controller and Playstation Move).The use of these non-traditional forms of inputs has led to the creation of new forms of gameplay and user feedback [1].However, this immersion can be broken in the case the players input is used in an improper way.Due to this fact, game developers and designers have to start looking for new and more reasonable ways for input devices [2].
In order to create games, the fun factor is an important feature to attract the player.Using a new device can create the opportunity to create new experiences to the player.Both interface and game must be carefully planed and tested, in order to provide a worthwhile experience, avoiding the frustration of the player.The use of neurofeedback in games or neurogaming can create a new territory in new entertainment field.
The electroencephalogram (EEG) signals are measures of voltage signals produced by neural activity.Nowadays, the systems that measure such data are becoming low cost and portable.In this work we use the Neurosky MindWave [3] since its one of the most popular.By using these devices, games need to have its gameplay redesigned with a new paradigm, including new types of challenges.Normally BCI based gameplay does not involve game mechanics' challenges but only BCI challenge, making games quite limited [4], [5].This work differs from those as it presents a traditional slice action game, called MindNinja, with a traditional input and a special input that comes from the brainwaves.Tests from this work show that the BCI could enhance the fun factor in this kind of game, making it possible to incorporate in others types of games.
Also, games that uses BCI can help children and persons with Attention deficit-hyperactivity disorder (ADHD) [6], which is also known as hyperkinetic disorder (HKD).ADHD is a neurobehavioral disorder that is characterised by either significant difficulties of maintaining attention for long periods of time or inappropriate impulsiveness and, in some cases, hyperactivity.The use of BCI to help children with ADHD has show some potential results [6], [7], and its usage together with games could lead to improvement in the attention level of persons with this disorder.
The BCI used by the authors is the NeuroSky MindWave that has up to 86% accuracy as show by the study [8].The user's Delta, Theta, Alpha, Beta, and Gamma brainwaves can be measured to detect the user's mental state, such as attention or relaxation, and thus be used as an input for a game connected to this device.Furthermore, and further to be researched, if users train certain brain patterns, more complex input information can be obtained.
The advance of mobile devices capacity allows them to be used for processing and providing physiological feedback.The use of the physiological monitoring paired with capabilities of the mobile device can be combined to deliver a game that can give the player an audio-visual feedback, which can promote activities that can help players detect their mental state.
The main goal of this project is to design, develop and evaluate a game that combines traditional game control elements (touch input) together with BCI.This project has decided to use a mobile platform using the touch screen as input, since it has a more natural interface [9].The main aspect of BCI used is the attention level, which affects the performance of the player in the game.To evaluate the game properly, this research has applied the MindNinja game with a controlled group of people.The main objective was to test the players user experience, usability, scores, and fun factor using this new approach.After playing the game with and without the device, players responded to an interview, and a questionnaire rating some characteristics from the game.These tests have shown that, by using the BCI the fun factor is higher.
We believe that this work not only introduces an architecture for this kind of games, but also rises important topics and considerations for the design of BCI based games.One important remark is that these games must be carefully designed for achieving a correct balance of the level of atention: games that require a constant and deep atention from the player may bring stress, while games that almost dont require any atention may achieve a difficult interaction balance.
This work is divided as follows: first the neurofeedback concepts are briefly presented, followed by the related work on the subject.Then, the design and some details of the implementation of the game are presented in Section 4. Section 5 presents the methodology that is used for the tests and section 6 the tests results.Finally section 7 presents the results.

II. NEUROFEEDBACK
The electro encephalogram or simply the EEG detect the electric activity from the user's brain and provides this information as a digital signal.EEG as a brain-computer interface (BCI) uses the EEG digital signals as input, making first a classification of them into categories.These categories could be used for command inputs of the game, such as the movement of a players or shooting enemies.
There are different methods to extract the intentions or thoughts of the users, like measurement of the brain activities over the motor cortex [10]; detection of periodic EEG waveforms patterns [11]; and identification of eventrelated potentials in the user's EEG waveforms that follows an event [12].
This work uses the NeuroSky Mindwave Mobile EEG headset, which is a minimally invasive and dry biosensor that can read electrical neuro-triggered activity in the brain to determine different brain waves and states.The reason this work has chosen this headset is because it has a low cost and it is also very easy to use BCI, but the principles present in this work could be adapted for others BCI devices.
There are also two mental stages that this work uses: attention and stress, which is calculated by the Neurosky SDK.This work uses the term attention referring to the capability of maintaining a selective concentration, focusing the mind on a single thought, task or object and stress as the response produced by the body when subjected to various types of physical or mental demand.The headset can also detect the blink of the eye.A report study from the company show that the headset is 86% accurate [8].But this device has some drawbacks, when compared with some others BCI devices, like its single channel and the input can have some noise.
Also engagement is a way to evaluate if the game is worth playing by the user.This work also uses the following formula for calculating a signal E of engagement based on alpha, beta, and theta waves that is highly correlated with participant task engagement [14]: which has been successfully used with Neurosky by [15].

III. RELATED WORKS
The application of a neurofeedback BCI technology can be applied by many areas, like communication [16], smart control of a house [17], [18] assistive technology [19], [20], [21] and gaming [5].With the BCI, the users attention level could be monitored, and new systems for focus could be built.The aim of this works is to investigate the potential of a BCI in games to provide help and a motivation for keeping the attention.This way the games could be useful for therapy of people with deficient of attention.
[22] present a study where the BCI of a student is monitored and a external agent check his attention level during a lesson.If the attention drops the agent try to recapture the diminishing attention levels by using verbal and nonverbal cues.
The BCI device is used to detect the players attention during the play of a FPS (first person shooter) game in the work [23].This is done in order to detect moments of the gameplay where the player was more attentive.On the same subject in [24], they evaluate the use of 3D stereo vision in a game, and the BCI is used to better evaluate the interaction with the game.Also in [25] students using a tutor system were evaluated using BCI, showing statical analysis that indicates that there is a relation between the lack of attention and the difficult of the student.In [26], authors describe users monitoring their attention while doing an exercise in the Second Life virtual world, showing that the users that are have more attention, performs the exercise better.
[27] presents the MindGame, a very simple game where the BCI is used for player movement in a 3D board.[28] presents MindTactics, a game where concentration and attention was used in a simple flag capture game.They have detect moments in the game where the player attention increases and where it decreases.In [5] a game controlled with some input from BCI and also with voice input is present.In this game the thought of the selection makes the selection.Results shows that this use of the BCI has a lot of errors (the selection is not made) and increases the frustration of the user.
In [4] shows a child game, where the player watches a story based on Hansel and Gretel, and from time to time they have to perform challenges with the mind in order to progress in the story.In [29] shows Mind maze, a game which experiences with BCI with mobile devices, similar to this work.The maze game has its gates open by the use of a certain pattern in the brainwaves, like attention or relaxation.Results from this work show that the fun factor has increased with the device.In [30] shows a attempt of building a system for including biofeedback in top off-the-self games.
BCI has also been applied in others areas of entertainment.In [33] presents a movie editor/viewer, which gather the users brain wave and put as an input to modify the current view movie, creating new movies.In [34] it is presented a study on the use of BCI devices for content creation.It show that the of the brain waves alone can create frustration on the users and other interfaces are also important.In [35] it is compared the usage of BCI for interacting with World of Warcraft game with traditional game interfaces.
In [36] it is presented a 3D environment where the user controls the movement of objects using the BCI.[37] and [38] uses a BCI to give feedback for a speech recognition system to detect missrecognition in order to improve the ASR.
[7], [6], [31] show studies of the use of BCI for treatment of ADHD.These works concludes that the use of neurofeedback training has effectively helped children with ADHD to increase the levels of attention during tasks.Also Asperger's Syndrome (AS) and Autistic Spectrum Disorder (ASD) can also gain from the use of neurofeedback games [32].

IV. THE ARCHITECTURE
Computer games are multimedia applications that employ knowledge of many different fields, such as Computer Graphics, Artificial Intelligence, Physics, Network and others [39].More, computer games are also interactive applications that exhibit three general classes of tasks: data acquisition, data processing, and data presentation [40], [41].Data acquisition in games is related to gathering data from input devices as keyboards, mice and joysticks.Data processing tasks consist on applying game rules, responding to user commands, simulating Physics and Artificial Intelligence behaviors.Data presentation tasks relate to providing feedback to the player about the current game state, usually through images and sound.This section will present how these tasks works together in the architecture.

A. The Architecture Overview
The architecture is composed of the the inputs, that can come in different forms, like: the touch screen (that can give single touch or gestures); the accelerometer; the microphone; gyroscope; compass; gps; and the BCI device.A graphics engine, that deals with all the render in the scene.A sound engine, that deals with all the sound management and configuration.And a statistics that deals with gathering data in order to provide data to be analysed.
This architecture uses a multithread game loop, where there is a main thread, responsible for the game, and a BCI thread responsible for dealing with the EEG brainwaves.The main thread is based on the single game loop, with a additional phase responsible for gathering messages from the other tread.First the game is initialised in a start phase, where all the resources are loaded, the bluetooth device is connected and the data are prepared for the beginning of the game.Then the main loop of the game happens, first the user input is gathered from the device in order to be processed in the update phase.Next a input manager phase, responsible for managing all the available inputs that can come from this thread or messages that comes from others threads and preparing them to the update phase.The update phase, where all the behavior, like physics and AI, and the required modification to the scene, according to the input, are processed according to a time step (the time elapsed since the last update).The last phase of the loop is the render, where the feedback is presented to the user, though images, sounds and vibrations.Also a final phase is required when the game is over, in order to unload all the data.
The aim of the proposed architecture is to provide an easy way to develop games with the use of BCI. Figure 1 illustrates the multithread game loop architecture of the game.This multithreaded architecture is composed by one thread for the main game loop (responsible for the traditional tasks of the game, like handling user input, presentation tasks and the update tasks), and another for the BCI module, which is responsible for gathering and processing the EEG brainwaves.Although the threads run independently from each other, a message is sent from the BCI to the main loop whenever new data is processed.
The BCI thread runs in a different thread, and is responsible for dealing with the all the data that comes from the BCI.At first this loop is responsible for pairing the device, in order to correct initialise and make sure that the bluetooth device is running perfectly.Then the BCI loop is started with signal acquisition phase, which gather all the brain signal data.This data is then processed by the signal processing phase, where the data is processed according to a heuristic, and then prepared to be sent as a message to the main loop.This message is prepared with all the data required to perform the actions on the main loop.The heuristic can be a simple formula or even a complex machine learning operation, like k-means [42].In the test case, simple formulas where used.All threads operations are non-blocking, where all the messages are implemented with a observer data pattern.
The architecture took as a base the Cocos2D framework [43] and the iPhone SDK.The Cocos2d is a 2D game library, which facilitates the development of games.The iPhone SDK is a development platform for iPhones, iPads and the iPod Touch.For game development it uses the objective-C language, has a 3D API based on OpenGL ES, and grants access for all the built-in hardware resources.Most of the change made by this framework in the Cocos2D rely on the input mechanism.

B. The Classes of the Architecture
This work has implemented some classes in the framework to handle the inputs and heuristics, as shown in the UML diagram, Figure 2.
The Input Task classes are responsible for handling user input.Input may come from different sources.In traditional desktop applications this corresponds to keyboards and mice, while mobile applications may use touch screens and microphones, for example.
The Input is a abstract class and is responsible for all the input that can affect the game.There are four types of inputs schemes handled by the framework: touch (handles input from touch screens); accelerometer (gathers input coming from accelerometer sensors); gesture (also from the touch screen, but in a form of gesture, like swap or pinch); GPS (gathers information coming from GPS sensors); Mic (gathers sounds that comes from the microphone); Compass (reads the device compass to determinate the direction of the player) and the EEG, which is the data from the BCI device retrieved by a bluetooth connection.The EEG also uses a special class called Bluetooth Initialiser Module, which is responsible for dealing with all the Bluetooth Initialisation, connection and management.Also there is the Heuristic Input, which process the BCI data accordantly to formulas or machine learning techniques.
The Heuristic classes and subclasses handle the BCI processing issues.This classes are responsible for the logic behind the processing of the received signal by the BCI.There are two main heuristics that can be implemented, a formula that simple process a mathematical formula according to the input, and a machine learning heuristic, that has access to the history data.The History Data is responsible for saving all the signals that comes from the BCI in order to give input to the machine learning algorithms and also can be used as statical data for a game developer.Also there is a special class, the Heuristic Manager, which is responsible for prepare and process all the available heuristics whenever needed, and also is responsible for saving the data in the History Data.
The Input Manager is a special class derived from the generic Input class.The Input Manager is responsible for instancing, managing, synchronising, and finalising all inputs used in the game.To guarantee the correct distribution and the abstraction of the tasks, the architecture defines a special task called Input Manager.The Input Manager is responsible for gather all the needed inputs, process and finally send them to the other tasks, like the Heuristic and the Statistics Manager.The Input Manager acts as a server and the inputs act as its clients.Every time a new input arrives, it sends a message to the Input Manager.The Input manager then sends this new input to the game thread, which will threat it accordingly.In order to guarantee central control of the inputs and their correct execution, there is only one instance of the Input Manager class in the application, implemented as the Singleton design pattern.Figure 3 illustrate the Input Manager and its interaction with the others classes.
The input subclasses send the data to the Input Manager when there is new information available.Then the Input Manager process this data and either transform them to commands for the Game Logic or send them to be processed by the Heuristics.The Heuristics receive the data and processes them according to the implemented heuristic.All the data is also sent to the Statistic Manager to be stored.• Inputs that affects the game (like the touch and BCI) and inputs that do not affect the game (like the accelerometer that could be used to see the preferred user's orientation of the device); • Data from the heuristics, achieved from their results and also time spent to process them; • Data from the game itself, such as points, lifes, misses and score; • Data from the hardware performance, suich as the FPS and time spent on each task;

C. The Architecture Execution
This subsection is dedicated to illustrate the execution of the architecture, so the reader can better understand it.Figure 4 illustrates the process.
The following execution workflow is used by the Input Manager to update the data from the BCI: first, it connects to the BCI device through the bluetooth; next, whenever the new data arrives from the BCI, the heuristic manager processes the data accordingly to the heuristics, save it on History Data and send it to the Input Manager as a message; Then the Input Manager prepare the game actions that has to be executed and send it to the update stage so that proper changes can be made.

V. GAME DESIGN OF THE TEST CASE
The paper presents the process of creating a new game prototype using EEG brainwaves as one of the players input.Although we present details of the specific game, we believe that our proposal may contribute to many different kinds of BCI based games.
One of the hypothesis of this work is that EEG signals must be combined with traditional interactions mechanics in order to be useful.The BCI interfaces may acquire mental states, but not details of what the user is thinking or imagining.Based on this limitation, the current paper proposes the design of a gameplay feedback composed by 2 categories: the mental state feedback and the mechanics feedback.
The proposed game is an action slice game, similar to Fruit Ninja [44], called MindNinja.The game play is simple: the player is a ninja that must slice the highest number of right objects while avoiding wrong objects.During the time limit of 60 seconds, the player slices the objects, by moving the finger on the touch screen, gaining points when correct objects are sliced and losing point when wrong objects, like a bomb, are sliced.While playing, the user must maintain his mental state as concentrated as possible.If his brainwaves show a cutback in attention, the game screen becomes foggy, making it more difficult to slice the objects (Figure 5 (a) shows the attention in 0 % and (b) at 50 % of attention).If the player can maintain his mental state in an attention mode at highest levels, everything in the game happens in slow motion (including the time), making it easier to slice the objects and achieving better scores as Figure 5 (c) shows.Therefore, in order to score higher points, the player needs This way the action game provides stimulation of the visual-motor skills like the fine motor skills and hand-eye coordination ability [45], in order to slice the objects.Also the game uses the visual stimuli in order to stimulate the perception ability since the user needs to be apple to detect bombs that should not be cut during the game.With the BCI, the game uses a extrinsic motivation for the user to keep its attention at a high level.The user can gain more points in the slow mode, since its easies to cut the right objects.

VI. METHODOLOGY
This work conducts a study to investigate the use of BCI for attention increase with the use of games.The game was tested with the BCI for five times by a group of eleven different users.Eleven subjects were recruited; 3 females and 8 males; ages ranged from 6 to 42.In this group two subjects have been diagnosed with ADHD.Also, there were different levels of experience with touch devices and mobile games in the group, ranging from low to high.None of the participants was physically disabled.
Table I shows the characteristics of the tested group.The study was performed by making each subject sit with the mobile phone in their hand and letting he play the game wearing the BCI device.All subjects were trained for five minutes by watching the observer playing the game and showing the gameplay.
Two types of results were gathered during the tests, a feedback from the users, and the game and brain statistics during the use of the game.All this tests were done with and without the BCI device affecting the gameplay.The brain and game statistics considered were:   the played session's.
• Missed Cuts: every time the player misses a correct object or cuts a bomb this number increases; • Attention Level: the mean attention level is gathered by the BCI during the users playing the game, ranging from 0 (no attention) to 100 (full focus); • Stress Level: the mean stress level is gathered by the BCI during the users playing the game, ranging from 0 (relaxed) to 100 (stressed); • Engagement Level:the mean engagement level is gathered by the BCI during the users' playing the game, ranging from 0 (not engaged) to 100 (engaged); • Evolution Attention:the attention recorded by the BCI during the last play divided by the first play time to measure its evolution.
The feedback was done by a series of questions made by the observer.These questions aim to provide a feedback from the user engagement, and the following characteristics were considered: • Ergonomic: the player will test the game with the BCI and without it and give it a grade in a scale ranging from 1 (very discomfortable) to 10 (very comfortable) of how he fells about its' comfortability; • Fun factor: the player will test the game with the BCI and without it and will grade the subjective fun from 1 (very boring) to 10 (very funny); • Difficult: the player will test the game with the BCI and without it and will grade 1 (very easy) to 10 (very difficult) the difficulty he had with it; • Feedback: the user will grade the experience he had with the BCI and without it and will grade it from 1 (bad) to 10 (great); • Time to learn: the observers will grade how difficult it was for the player to learn the gameplay with the BCI and without it, grading it from 1 (very easy) to 10 (very difficult);

VII. RESULTS
In order to evaluate our architecture, the Apple's iPhone 4S mobile device was used, which is equipped with an ARM A5 800 MHz CPU, 512 MB of RAM, touch screen, accelerometer, bluetooth, and wiFi equipped with the Neu-roSky Mindwave Mobile EEG headset.
In order to test the game architecture performance, we executed our tests in two different stages.The first one occurs at the beginning of a game session, before the screen is displayed.The second stage occurs when the player is actually playing the game.Table II illustrates these tests.To assure that results are consistent, all tests of this work were repeated 10 times and the standard deviation of the average times was confirmed to be within 5%.The results describe that the "initialisation" stage takes more time to run than the other stages.This happens since it is necessary to initialise the BCI and also to load the resource.The "In Game" stages runs very smoothly, having a performance of more than 65 FPS.
Table III shows the statical results of all participants and also the average and standard deviation from these results.
This tests show that most of the subjects have more points playing and also less erros with the BCI, but that cannot be put into account, since the BCI affects the gameplay.Subjects with higher attention levels had the highest scores (with ou without the BCI affecting the gameplay).Most the subjects without the BCI and all with the BCI had some increase in the attention level, as comparing the first time they play the game with the fifth time they play the game.Also most players with the BCI had a more engaging experience when compared without the BCI.All the players that had the BCI affecting the gameplay had a more expressive evolution of the attention level, like Figure 6 illustrates.
As an example, Figure 7 shows the attention data from user E in two conducted tests, the first time he played the game and the fifth time he had played the game, in order to see how his attention level behave during a game.
Figure 8 shows the attention level from user E in two different tests, one with BCI and another with traditional interfaces.In order to test in a situation that he already knows how to play, the measurement corresponds to the 5th time he plays the game.From these results it is possible to see that without the attention affection the gameplay, the attention level is very low, since its his fifth time he plays the game, and he does not need a high level of attention.This results show that the BCI affecting in the gameplay can retain the attention of the player for more time.
Table IV shows the ergonomic results of all participants and also the average and standard deviation from these results.
Even thought users prefer are more comfortable without the BCI device, tests show that most of the subjects felt comfortable using the headset and all the subjects have the opinion that the BCI game is more fun, and they had a good experience playing with it.Some subjects had some difficulties with the game (subject C and F), one subject had difficult with the game with the BCI (Subject H) and two had more difficulty learning its usage (subject C and F).VIII.CONCLUSION New forms of user inputs are being researched by industry in order to attract more players and enhance players immersion during game play.This work has proposed an architecture for developing 2D games with the BCI.In order to validate it, this works shows the design, development and evaluation process of a simple game to be played using the BCI input system.Tests shows that the BCI equipment could increase the immersion, showing great potential to enhance the fun level of the game.Also tests done in this work indicate that this kind of games could help people with low attention to gain more control over its attention level, but tests with a bigger group should be done to validate this claim.
The main goal of this work was to research the potential of the use of EEG devices in conjunction with games, both in terms of introducing a new input device and new possibilities of human-computer interaction with a new form of interaction using the brainwave.The tests and analysis of the MindNinja gave valuable insight information for further investigation and development of new strategies for the use of BCI in games.
Future work will concentrate in providing more types of games, more tests of others types of cognitive behavior during the game and also further development of the framework to identify more brain patterns.And also, future work will try to use the BCI with a machine learning algorithm.

Figure 1 :
Figure 1: Workflow of the game loop.

Figure 2 :
Figure 2: UML of the Input and Heuristics Classes.

Figure 3 :
Figure 3: The Input Manager Process.

Figure 4 :
Figure 4: Execution of the Input Manager.

Figure 5 :
Figure 5: The MindNinja Game With different Attention Levels

Figure 6 :
Figure 6: Evolution of Attention level of User Group comparing the first and last tests.

Figure 7 :
Figure 7: Attention level of User E during the in the first and last tests.

Figure 8 :
Figure 8: Attention level of User E during the in the first and last tests.

Table I :
Test Group Characteristics.

Table II :
Numerical results from the game on different steps.

Table III :
Results of the usability tests Without the BCI.