Model, Taxonomy and Methodology for Research Employing EEG-based Brain-Computer Interface Games
Resumo
The rapid expansion of Brain-Computer Interface technology, aligned with the advancements on the fields of Human-Computer Interaction, Physiological Computing and Machine Learning, allowed for the recent development of applications outside of clinical environments, such as education, arts and games. Games controlled by electroencephalography (EEG), a specific case of BCI technology, benefit from both the fields of BCI and games, since they can be played by virtually any person regardless of physical condition, can be applied in numerous contexts, and are ludic by nature. Despite these recent advancements, there is still no solid theoretical foundation to aggregate the terminology and methods of these fields, since current models and classification schemes can represent characteristics of either BCI systems or games, but not both. In this sense, the thesis summarized in this work presents a model for representing EEG-based BCI games, a taxonomy for classifying and comparing studies of the field, and a methodology for conducting scientific studies using those games. The model is intended to describe and develop new EEG games by instantiating its components. The CoDIS taxonomy considers four aspects of such games: concept, design, implementation and study, each with different dimensions to represent various of their characteristics. Based on the model and the taxonomy, the PIERSE methodology was developed for the planning, implementation, execution and reporting of scientific experiments using those games. The contributions of the thesis are detailed in various dimensions in this manuscript.
Palavras-chave:
HCI, BCI, EEG, Games
Referências
Abdulla Ali and Sadasivan Puthusserypady. 2015. A 3D learning playground for potential attention training in ADHD: A brain computer interface approach. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE. DOI: 10.1109/embc.2015.7318302
Alissa N. Antle, Leslie Chesick, and Elgin-Skye Mclaren. 2018. Opening up the design space of neurofeedback Brain-Computer Interfaces for children. ACM Transactions on Computer-Human Interaction 24, 6 (Jan. 2018), 1–33. DOI: 10.1145/3131607
H. Banville and T.H. Falk. 2016. Recent advances and open challenges in hybrid brain-computer interfacing: A technological review of non-invasive human research. Brain-Computer Interfaces 3, 1 (Jan. 2016), 9–46. DOI: 10.1080/2326263x.2015.1134958
M. A. Bento, L. C. Miranda, and G. A. M. Vasiljevic. 2022. Player-Game Interaction Recorder: Um Software de Gravação para Apoiar a Avaliação da Interação de Jogadores com Jogos Digitais. In 21st Brazilian Symposium on Computer Games and Digital Entertainment. 1177–1186. DOI: 10.5753/sbgames_estendido.2022.226079
Sasha Burwell, Matthew Sample, and Eric Racine. 2017. Ethical aspects of brain computer interfaces: a scoping review. BMC Medical Ethics 18, 1 (Nov. 2017). DOI: 10.1186/s12910-017-0220-y
Inchul Choi, Ilsun Rhiu, Yushin Lee, Myung Hwan Yun, and Chang S. Nam. 2017. A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives. PLOS ONE 12, 4 (April 2017), 1–35. DOI: 10.1371/journal.pone.0176674
Alessandro Luiz Stamatto Ferreira, Leonardo Cunha de Miranda, Erica Esteves Cunha de Miranda, and Sarah Gomes Sakamoto. 2013. A survey of interactive systems based on Brain-Computer Interfaces. SBC Journal on 3D Interactive Systems 4, 1 (2013), 3–13.
Alessandro Luiz Stamatto Ferreira, Juvane Nunes Marciano, Leonardo Cunha de Miranda, and Erica Esteves Cunha de Miranda. 2014. Understanding and proposing a design rationale of digital games based on Brain-Computer Interface: Results of the AdmiralMind Battleship study. SBC Journal on Interactive Systems 5, 1 (2014), 3–15. DOI: 10.5753/jis.2014.638
Camille Jeunet, Stefan Debener, Fabien Lotte, Jérémie Mattout, Reinhold Scherer, and Catharina Zich. 2018. Mind the traps: Design guidelines for rigorous BCI experiments (1st Edition ed.). Taylor & Francis, Chapter 32, 1–22.
A. Y. Kaplan, S. L. Shishkin, I. P. Ganin, I. A. Basyul, and A. Y. Zhigalov. 2013. Adapting the P300-based Brain–Computer Interface for gaming: A review. IEEE Transactions on Computational Intelligence and AI in Games 5, 2 (June 2013), 141–149. DOI: 10.1109/TCIAIG.2012.2237517
Bojan Kerous, Filip Skola, and Fotis Liarokapis. 2018. EEG-based BCI and video games: A progress report. Virtual Reality 22, 2 (2018), 119–135. DOI: 10.1007/s10055-017-0328-x
B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28, 8 (Aug. 2002), 721–734. DOI: 10.1109/tse.2002.1027796
Nataliya Kosmyna and Anatole Lécuyer. 2019. A conceptual space for EEG-based brain-computer interfaces. PLOS ONE 14, 1 (01 2019), 1–30. DOI: 10.1371/journal.pone.0210145
Ioulietta Lazarou, Spiros Nikolopoulos, Panagiotis C. Petrantonakis, Ioannis Kompatsiaris, and Magda Tsolaki. 2018. EEG-based brain–computer interfaces for communication and rehabilitation of people with motor impairment: A novel approach of the 21st century. Frontiers in Human Neuroscience 12 (2018), 14. DOI: 10.3389/fnhum.2018.00014
Scott T. Leatherdale. 2018. Natural experiment methodology for research: A review of how different methods can support real-world research. International Journal of Social Research Methodology 22, 1 (July 2018), 19–35. DOI: 10.1080/13645579.2018.1488449
Mikhail A. Lebedev and Miguel A.L. Nicolelis. 2006. Brain–machine interfaces: Past, present and future. Trends in Neurosciences 29, 9 (Sept. 2006), 536–546. DOI: 10.1016/j.tins.2006.07.004
F. M. C. Lima, L. C. Miranda, G. A. M. Vasiljevic, and M. C. C. Baranauskas. 2019. An Intra and Inter-conferences Correlation Analysis of the Field of HCI: Revealing New Trends from the International and Brazilian Communities. In 18th Brazilian Symposium on Human Factors in Computing Systems. 1–11. DOI: 10.1145/3357155.3358476
F. M. C. Lima, G. A. M. Vasiljevic, L. C. Miranda, and M. C. C. Baranauskas. 2021. An Analysis of IHC and HCII Publication Titles: Revealing and Comparing the Topics of Interest of their Communities. Journal on Interactive Systems 12, 1 (March 2021), 1–20. DOI: 10.5753/jis.2021.997
David Marshall, Damien Coyle, Shane Wilson, and Michael Callaghan. 2013. Games, gameplay, and BCI: The state of the art. IEEE Transactions on Computational Intelligence and Al in Games 5, 2 (2013), 82–99.
S. G. Mason and G. E. Birch. 2003. A general framework for brain-computer interface design. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11, 1 (March 2003), 70–85. DOI: 10.1109/TNSRE.2003.810426
S. G. Mason, M. M. Moore Jackson, and G. E. Birch. 2005. A general framework for characterizing studies of Brain Interface technology. Annals of Biomedical Engineering 33, 11 (Nov. 2005), 1653–1670. DOI: 10.1007/s10439-005-7706-3
Luis F. Nicolas-Alonso and Jaime Gomez-Gil. 2012. Brain-Computer Interfaces, a review. Sensors 12, 2 (2012), 1211–1279.
Yunsick Sung, Kyungeun Cho, and Kyhyun Um. 2012. A development architecture for serious games using BCI (Brain Computer Interface) sensors. Sensors 12, 11 (Nov. 2012), 15671–15688. DOI: 10.3390/s121115671
G. A. M. Vasiljevic and L. C. Miranda. 2019. The effect of auditory stimuli on user’s meditation and workload in a brain-computer interface game. Interacting with Computers 31, 3 (2019), 250–262. DOI: 10.1093/iwc/iwz014
G. A. M. Vasiljevic and L. C. Miranda. 2019. The influence of graphical elements on user’s attention and control on a neurofeedback-based game. Entertainment Computing 29 (2019), 10–19. DOI: 10.1016/j.entcom.2018.10.003
G. A. M. Vasiljevic and L. C. Miranda. 2020. Brain–computer interface games based on consumer-grade EEG devices: A systematic literature review. International Journal of Human-Computer Interaction 36, 2 (2020), 105–142. DOI: 10.1080/10447318.2019.1612213
G. A. M. Vasiljevic and L. C. Miranda. 2020. A general model for electroencephalography-controlled brain-computer interface games. In 20th International Conference on Computational Science and Its Applications. 174–189. DOI: 10.1007/978-3-030-58802-1_13
G. A. M. Vasiljevic and L. C. Miranda. 2022. Comparing users’ performance and game experience between a competitive and collaborative brain-computer interface. Behaviour & Information Technology (2022), 1–20. DOI: 10.1080/0144929X.2022.2152727
G. A. M. Vasiljevic and L. C. Miranda. 2023. The CoDIS Taxonomy for Brain-Computer Interface Games controlled by Electroencephalography. International Journal of Human-Computer Interaction (2023), 1–28.
G. A. M. Vasiljevic, L. C. Miranda, and B. C. Menezes. 2018. Mental War: An attention-based single/multiplayer brain-computer interface game. In 18th International Conference on Computational Science and Its Applications. 450–465. DOI: 10.1007/978-3-319-95162-1_31
G. A. M. Vasiljevic, L. C. Miranda, and B. C. Menezes. 2018. Zen Cat: A meditation-based brain-computer interface game. In 18th International Conference on Computational Science and Its Applications. 294–309. DOI: 10.1007/978-3-319-95162-1_20
G. A. M. Vasiljevic, L. C. Miranda, and E. E. C. Miranda. 2014. MasterMind Chess: Design and implementation of classic, Capablanca and Fischer modes with real time match observation. In 13th Brazilian Symposium on Computer Games and Digital Entertainment. 1–9.
G. A. M. Vasiljevic, L. C. Miranda, and E. E. C. Miranda. 2016. A case study of MasterMind Chess: Comparing mouse/keyboard interaction with Kinect-based gestural interface. Advances in Human-Computer Interaction 2016, 1 (2016), 1–10.
G. A. M. Vasiljevic, L. C. Miranda, E. E. C. Miranda, and L. F. Silva. 2012. Prototipação de interfaces tangíveis de produtos interativos: Estado da arte e desafios da plataforma Arduino. In Proceedings of the XI Brazilian Symposium on Human Factors in Computing Systems(ACM International Conference Proceeding Series). SBC, Cuiabá, MT, 1–10.
Yijun Wang, Masaki Nakanishi, and Dan Zhang. 2019. EEG-based brain-computer interfaces. In Advances in Experimental Medicine and Biology. Springer Singapore, 41–65. DOI: 10.1007/978-981-13-2050-7_2
Jonathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert Pfurtscheller, and Theresa M. Vaughan. 2002. Brain-computer interfaces for communication and control. Clinical Neurophysiology 113, 6 (2002), 767–791.
Thorsten O. Zander, Christian Kothe, S. Welke, and M. Roetting. 2009. Utilizing secondary input from passive brain-computer interfaces for enhancing human-machine interaction. In Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience. Vol. 5638 LNCS. DOI: 10.1007/978-3-642-02812-0_86
Alissa N. Antle, Leslie Chesick, and Elgin-Skye Mclaren. 2018. Opening up the design space of neurofeedback Brain-Computer Interfaces for children. ACM Transactions on Computer-Human Interaction 24, 6 (Jan. 2018), 1–33. DOI: 10.1145/3131607
H. Banville and T.H. Falk. 2016. Recent advances and open challenges in hybrid brain-computer interfacing: A technological review of non-invasive human research. Brain-Computer Interfaces 3, 1 (Jan. 2016), 9–46. DOI: 10.1080/2326263x.2015.1134958
M. A. Bento, L. C. Miranda, and G. A. M. Vasiljevic. 2022. Player-Game Interaction Recorder: Um Software de Gravação para Apoiar a Avaliação da Interação de Jogadores com Jogos Digitais. In 21st Brazilian Symposium on Computer Games and Digital Entertainment. 1177–1186. DOI: 10.5753/sbgames_estendido.2022.226079
Sasha Burwell, Matthew Sample, and Eric Racine. 2017. Ethical aspects of brain computer interfaces: a scoping review. BMC Medical Ethics 18, 1 (Nov. 2017). DOI: 10.1186/s12910-017-0220-y
Inchul Choi, Ilsun Rhiu, Yushin Lee, Myung Hwan Yun, and Chang S. Nam. 2017. A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives. PLOS ONE 12, 4 (April 2017), 1–35. DOI: 10.1371/journal.pone.0176674
Alessandro Luiz Stamatto Ferreira, Leonardo Cunha de Miranda, Erica Esteves Cunha de Miranda, and Sarah Gomes Sakamoto. 2013. A survey of interactive systems based on Brain-Computer Interfaces. SBC Journal on 3D Interactive Systems 4, 1 (2013), 3–13.
Alessandro Luiz Stamatto Ferreira, Juvane Nunes Marciano, Leonardo Cunha de Miranda, and Erica Esteves Cunha de Miranda. 2014. Understanding and proposing a design rationale of digital games based on Brain-Computer Interface: Results of the AdmiralMind Battleship study. SBC Journal on Interactive Systems 5, 1 (2014), 3–15. DOI: 10.5753/jis.2014.638
Camille Jeunet, Stefan Debener, Fabien Lotte, Jérémie Mattout, Reinhold Scherer, and Catharina Zich. 2018. Mind the traps: Design guidelines for rigorous BCI experiments (1st Edition ed.). Taylor & Francis, Chapter 32, 1–22.
A. Y. Kaplan, S. L. Shishkin, I. P. Ganin, I. A. Basyul, and A. Y. Zhigalov. 2013. Adapting the P300-based Brain–Computer Interface for gaming: A review. IEEE Transactions on Computational Intelligence and AI in Games 5, 2 (June 2013), 141–149. DOI: 10.1109/TCIAIG.2012.2237517
Bojan Kerous, Filip Skola, and Fotis Liarokapis. 2018. EEG-based BCI and video games: A progress report. Virtual Reality 22, 2 (2018), 119–135. DOI: 10.1007/s10055-017-0328-x
B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28, 8 (Aug. 2002), 721–734. DOI: 10.1109/tse.2002.1027796
Nataliya Kosmyna and Anatole Lécuyer. 2019. A conceptual space for EEG-based brain-computer interfaces. PLOS ONE 14, 1 (01 2019), 1–30. DOI: 10.1371/journal.pone.0210145
Ioulietta Lazarou, Spiros Nikolopoulos, Panagiotis C. Petrantonakis, Ioannis Kompatsiaris, and Magda Tsolaki. 2018. EEG-based brain–computer interfaces for communication and rehabilitation of people with motor impairment: A novel approach of the 21st century. Frontiers in Human Neuroscience 12 (2018), 14. DOI: 10.3389/fnhum.2018.00014
Scott T. Leatherdale. 2018. Natural experiment methodology for research: A review of how different methods can support real-world research. International Journal of Social Research Methodology 22, 1 (July 2018), 19–35. DOI: 10.1080/13645579.2018.1488449
Mikhail A. Lebedev and Miguel A.L. Nicolelis. 2006. Brain–machine interfaces: Past, present and future. Trends in Neurosciences 29, 9 (Sept. 2006), 536–546. DOI: 10.1016/j.tins.2006.07.004
F. M. C. Lima, L. C. Miranda, G. A. M. Vasiljevic, and M. C. C. Baranauskas. 2019. An Intra and Inter-conferences Correlation Analysis of the Field of HCI: Revealing New Trends from the International and Brazilian Communities. In 18th Brazilian Symposium on Human Factors in Computing Systems. 1–11. DOI: 10.1145/3357155.3358476
F. M. C. Lima, G. A. M. Vasiljevic, L. C. Miranda, and M. C. C. Baranauskas. 2021. An Analysis of IHC and HCII Publication Titles: Revealing and Comparing the Topics of Interest of their Communities. Journal on Interactive Systems 12, 1 (March 2021), 1–20. DOI: 10.5753/jis.2021.997
David Marshall, Damien Coyle, Shane Wilson, and Michael Callaghan. 2013. Games, gameplay, and BCI: The state of the art. IEEE Transactions on Computational Intelligence and Al in Games 5, 2 (2013), 82–99.
S. G. Mason and G. E. Birch. 2003. A general framework for brain-computer interface design. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11, 1 (March 2003), 70–85. DOI: 10.1109/TNSRE.2003.810426
S. G. Mason, M. M. Moore Jackson, and G. E. Birch. 2005. A general framework for characterizing studies of Brain Interface technology. Annals of Biomedical Engineering 33, 11 (Nov. 2005), 1653–1670. DOI: 10.1007/s10439-005-7706-3
Luis F. Nicolas-Alonso and Jaime Gomez-Gil. 2012. Brain-Computer Interfaces, a review. Sensors 12, 2 (2012), 1211–1279.
Yunsick Sung, Kyungeun Cho, and Kyhyun Um. 2012. A development architecture for serious games using BCI (Brain Computer Interface) sensors. Sensors 12, 11 (Nov. 2012), 15671–15688. DOI: 10.3390/s121115671
G. A. M. Vasiljevic and L. C. Miranda. 2019. The effect of auditory stimuli on user’s meditation and workload in a brain-computer interface game. Interacting with Computers 31, 3 (2019), 250–262. DOI: 10.1093/iwc/iwz014
G. A. M. Vasiljevic and L. C. Miranda. 2019. The influence of graphical elements on user’s attention and control on a neurofeedback-based game. Entertainment Computing 29 (2019), 10–19. DOI: 10.1016/j.entcom.2018.10.003
G. A. M. Vasiljevic and L. C. Miranda. 2020. Brain–computer interface games based on consumer-grade EEG devices: A systematic literature review. International Journal of Human-Computer Interaction 36, 2 (2020), 105–142. DOI: 10.1080/10447318.2019.1612213
G. A. M. Vasiljevic and L. C. Miranda. 2020. A general model for electroencephalography-controlled brain-computer interface games. In 20th International Conference on Computational Science and Its Applications. 174–189. DOI: 10.1007/978-3-030-58802-1_13
G. A. M. Vasiljevic and L. C. Miranda. 2022. Comparing users’ performance and game experience between a competitive and collaborative brain-computer interface. Behaviour & Information Technology (2022), 1–20. DOI: 10.1080/0144929X.2022.2152727
G. A. M. Vasiljevic and L. C. Miranda. 2023. The CoDIS Taxonomy for Brain-Computer Interface Games controlled by Electroencephalography. International Journal of Human-Computer Interaction (2023), 1–28.
G. A. M. Vasiljevic, L. C. Miranda, and B. C. Menezes. 2018. Mental War: An attention-based single/multiplayer brain-computer interface game. In 18th International Conference on Computational Science and Its Applications. 450–465. DOI: 10.1007/978-3-319-95162-1_31
G. A. M. Vasiljevic, L. C. Miranda, and B. C. Menezes. 2018. Zen Cat: A meditation-based brain-computer interface game. In 18th International Conference on Computational Science and Its Applications. 294–309. DOI: 10.1007/978-3-319-95162-1_20
G. A. M. Vasiljevic, L. C. Miranda, and E. E. C. Miranda. 2014. MasterMind Chess: Design and implementation of classic, Capablanca and Fischer modes with real time match observation. In 13th Brazilian Symposium on Computer Games and Digital Entertainment. 1–9.
G. A. M. Vasiljevic, L. C. Miranda, and E. E. C. Miranda. 2016. A case study of MasterMind Chess: Comparing mouse/keyboard interaction with Kinect-based gestural interface. Advances in Human-Computer Interaction 2016, 1 (2016), 1–10.
G. A. M. Vasiljevic, L. C. Miranda, E. E. C. Miranda, and L. F. Silva. 2012. Prototipação de interfaces tangíveis de produtos interativos: Estado da arte e desafios da plataforma Arduino. In Proceedings of the XI Brazilian Symposium on Human Factors in Computing Systems(ACM International Conference Proceeding Series). SBC, Cuiabá, MT, 1–10.
Yijun Wang, Masaki Nakanishi, and Dan Zhang. 2019. EEG-based brain-computer interfaces. In Advances in Experimental Medicine and Biology. Springer Singapore, 41–65. DOI: 10.1007/978-981-13-2050-7_2
Jonathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert Pfurtscheller, and Theresa M. Vaughan. 2002. Brain-computer interfaces for communication and control. Clinical Neurophysiology 113, 6 (2002), 767–791.
Thorsten O. Zander, Christian Kothe, S. Welke, and M. Roetting. 2009. Utilizing secondary input from passive brain-computer interfaces for enhancing human-machine interaction. In Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience. Vol. 5638 LNCS. DOI: 10.1007/978-3-642-02812-0_86
Publicado
16/10/2023
Como Citar
VASILJEVIC, Gabriel Alves Mendes; CUNHA DE MIRANDA, Leonardo.
Model, Taxonomy and Methodology for Research Employing EEG-based Brain-Computer Interface Games. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
