Visualization of brainwaves using EEG to map emotions with eye tracking to identify attention in audiovisual workpieces

  • Thiago Henrique Coelho Tavares Da Silva UFPB
  • Matheus Dantas Cavalcanti UFPB
  • Felipe Melo Feliciano De Sá UFPB
  • Isaac Nóbrega Marinho UFPB
  • Daniel De Queiroz Cavalcanti UFPB
  • Valdecir Becker UFPB

Resumo


This article describes a brainwave visualization system using EEG and Eye Tracking in order to map the emotional relationship of individuals with audiovisual workpieces, especially attention and taste. Using the Design Science Research method, the artifact was specified, implemented and tested with 10 subjects, using a horror movie trailer. A preliminary and a post-test questionnaire was presented to the participants. The results indicate patterns of emotional identification with the film, which can be interpreted as an inclination to watch the film in movie theaters or a repulsion to the theme/genre of the film. In conclusion, this research points to an advance in evaluation of audiovisual workpieces, contemplating unconscious emotional elements of subjective perceptions about the watched content.

Palavras-chave: Brain-Computer Interfaces, EEG, Human-Computer interaction, emotions, Eye Tracking

Referências

Sarah N. Abdulkader, Ayman Atia, and Mostafa-Sami M. Mostafa. 2015. Brain computer interfacing: Applications and challenges. Egyptian Informatics Journal 16, 2 (2015), 213–230. https://doi.org/10.1016/j.eij.2015.06.002

Cassidy AndrewJoseph. 2019. What is the sampling rate for the INSIGHT and why has it been designed this way? [link]. 8 de junho de 2022

Bruce L. Archer. 1964. Systematic method for designers. Design (1964), 56–59.

André Barbosa, Bryan Souza, Antonio Pereira, and Adelardo Medeiros. 2009. IMPLEMENTAÇÃO DE CLASSIFICADOR DE TAREFAS MENTAIS BASEADO EM EEG. In Anais do 9 Congresso Brasileiro de Redes Neurais. SBRN, Ouro Preto, MG, 1–5. https://doi.org/10.21528/CBRN2009-152

Valdecir Becker, Matheus Cavalcanti, Thiago Silva, Edvaldo Vasconcelos, Alessandro Pinon, and Felipe Melo. 2022. A System for Graphical Visualization of Brainwaves to Analyse Media Content Consumption. In Human-Computer Interaction. Technological Innovation, Masaaki Kurosu (Ed.). Springer International Publishing, Cham, 318–328.

Valdecir Becker, Daniel Gambaro, Thais Saraiva Ramos, and Rafael Moura Toscano. 2018. Audiovisual Design: Introducing ‘Media Affordances’ as a Relevant Concept for the Development of a New Communication Model. In Applications and Usability of Interactive Television, María José Abásolo, Jorge Abreu, Pedro Almeida, and Telmo Silva (Eds.). Springer International Publishing, Cham, 17–31.

Valdecir Becker, Thiago Silva, Matheus Cavalcanti, Daniel Gambaro, and Jordan Elias. 2021. Potencial das interfaces cérebro máquina para a recomendação de conteúdos em sistemas de vídeo sob demanda. In 4o Congresso Internacional Media Ecology and Image Studies - Reflexões sobre o ecossistema midiático pós pandemia. Ria Editorial, 145–167. [link]

Valdecir Becker, Rafael Toscano, Amanda Azevedo, and Daniel Gambaro. 2019. The Concept of Interaction Triggers in Audiovisual Design Model and Its Application to Develop an Interactive Museum. In Applications and Usability of Interactive TV, María José Abásolo, Telmo Silva, and Nestor D. González (Eds.). Springer International Publishing, Cham, 28–39.

Tanja Blascheck, Kuno Kurzhals, Michael Raschke, Michael Burch, Daniel Weiskopf, and Thomas Ertl. 2014. State-of-the-Art of Visualization for Eye Tracking Data. In EuroVis - STARs, R. Borgo, R. Maciejewski, and I. Viola (Eds.). The Eurographics Association. https://doi.org/10.2312/eurovisstar.20141173

Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (1994), 49–59. https://doi.org/10.1016/0005-7916(94)90063-9

Blain Brown. 2016. Cinematography: Theory and Practice Image Making for Cinematographers and Directors. Focal Press, Waltham, MA, USA.

Ruth M. J. Byrne. 2015. Mental Models. John Wiley & Sons, Ltd, 1–13. https://doi.org/10.1002/9781118900772.etrds0217 arXiv: [link]

Vamsi Dattada, Vijay Mohan and M Jeevan. 2019. Analysis of Concealed Anger Emotion in a Neutral Speech Signal. In 2019 IEEE International Conference on Distributed Computing, VLSI, Electrical Circuits and Robotics (DISCOVER). 1–5. https://doi.org/10.1109/DISCOVER47552.2019.9008037

Aline Dresch, Daniel Lacerda, and Junico Antunes. 2015. Design Science Research: Método de Pesquisa para Avanço da Ciência e Tecnologia. 198 pages. https://doi.org/10.13140/2.1.2264.2885

Emotiv. 2021. Data Subscription. https://emotiv.gitbook.io/cortex-api/data-subscription 8 de junho de 2022

Emotiv. 2022. Connecting to the Cortex API. https://emotiv.gitbook.io/cortex-api/connecting-to-the-cortex-api 8 de junho de 2022

Emotiv. 2022. Cortex API Getting Started. https://emotiv.gitbook.io/cortex-api/ 8 de junho de 2022

Emotiv. 2022. Insight Manual Technichal Specification. https://emotiv.gitbook.io/insight-manual/introduction/technical-specifications 8 de junho de 2022

Bryn Farnsworth. 2020. 10 Most Used Eye Tracking Metrics and Terms. https://imotions.com/blog/10-terms-metrics-eye-tracking/ 6 de junho de 2022

Bryn Fransworth. 2019. EEG (Electroencephalography): The Complete Pocket Guide. https://imotions.com/blog/eeg/ 6 de junho de 2022

Jérémy Frey, Jelena Mladenovic, Fabien Lotte, Camille Jeunet, and Léa Pillette. 2017. When HCI Meets Neurotechnologies: What You Should Know about Brain-Computer Interfaces. 1253–1256. https://doi.org/10.1145/3027063.3027100

Crystal Gabert-Quillen, Ellen Bartolini, Benjamin Swerdlow, and Charles Sanislow. 2014. Ratings for emotion film clips. Behavior research methods 47 (07 2014), 773–787. https://doi.org/10.3758/s13428-014-0500-0

Zhen Gao and Shangfei Wang. 2015. Emotion Recognition from EEG Signals Using Hierarchical Bayesian Network with Privileged Information. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval (Shanghai, China) (ICMR ’15). Association for Computing Machinery, New York, NY, USA, 579–582. https://doi.org/10.1145/2671188.2749364

James J. Gross and Robert W. Levenson. 1995. Emotion elicitation using films. Cognition and Emotion 9, 1 (1995), 87–108. https://doi.org/10.1080/02699939508408966

Alan R. Hevner, Salvatore T. March, Jinsoo Park, and Sudha Ram. 2004. Design Science in Information Systems Research. MIS Q. 28, 1 (mar 2004), 75–105.

Alan R. Hevner and Harlan D. Mills. 1995. Box-structured requirements determination methods. Decision Support Systems 13, 3 (1995), 223–239. Information technologies and systems. https://doi.org/10.1016/0167-9236(94)E0044-R

Pertti Järvinen. 2007. Action Research is Similar to Design Science. Quality and Quantity 41 (02 2007), 37–54. https://doi.org/10.1007/s11135-005-5427-1

João Damasceno Martins Ladeira. 2019. O algoritmo e o fluxo: Netflix, aprendizado de máquina e algoritmos de recomendações. Intexto 47 (ago. 2019), 166–184. https://doi.org/10.19132/1807-8583201947.166-184

Yang Li, Wenming Zheng, Yuan Zong, Zhen Cui, Tong Zhang, and Xiaoyan Zhou. 2021. A Bi-Hemisphere Domain Adversarial Neural Network Model for EEG Emotion Recognition. IEEE Transactions on Affective Computing 12, 2 (2021), 494–504. https://doi.org/10.1109/TAFFC.2018.2885474

Yifei Lu, Wei-Long Zheng, Binbin Li, and Bao-Liang Lu. 2015. Combining Eye Movements and EEG to Enhance Emotion Recognition. In Proceedings of the 24th International Conference on Artificial Intelligence (Buenos Aires, Argentina) (IJCAI’15). AAAI Press, 1170–1176.

Juan-Miguel López-Gil, Jordi Virgili-Gomá, Rosa Gil, Teresa Guilera, Iolanda Batalla, Jorge Soler-González, and Roberto García. 2016. Method for Improving EEG Based Emotion Recognition by Combining It with Synchronized Biometric and Eye Tracking Technologies in a Non-invasive and Low Cost Way. Frontiers in Computational Neuroscience 10 (2016). https://doi.org/10.3389/fncom.2016.00085

Salvatore March and Gerald Smith. 1995. Design and Natural Science Research on Information Technology. Decision Support Systems 15 (12 1995), 251–266. https://doi.org/10.1016/0167-9236(94)00041-2

Bruna Maiara Xavier Mariano. 2015. Produção, distribuição e interação: um estudo sobre o Netflix e a nova dinâmica de consumo audiovisual. (2015).

Joseph V. Mascelli. 2010. Os Cinco Cs da Cinematografia. Summus, São Paulo, SP, Brasil.

Wagner Rodrigues Miranda. 2017. Netflix: Big Data e os algoritmos de recomendação1. Anais do XXXII INTERCOM, Rio de Janeiro (2017).

Gelareh Mohammadi and Patrik Vuilleumier. 2020. A Multi-Componential Approach to Emotion Recognition and the Effect of Personality. IEEE Transaction on Affective Computing (2020), 1–1. https://doi.org/10.1109/TAFFC.2020.3028109

Garibaldi Greice Mussatto and Scheila de Avila e Silva. 2014. Perspectivas e Potencialidades da Interface Cérebro-Máquina. Revista de Sistemas de Informação da FSMA 13, Article 5 (Jan-Jun 2014), 6 pages. http://www.fsma.edu.br/si/edicao13/FSMA_SI_2014_1_Estudantil_3.html

Alex Poole and Ball J. Linden. 2005. Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future. In Prospects”, Chapter in C. Ghaoui (Ed.): Encyclopedia of Human-Computer Interaction. Pennsylvania: Idea Group, Inc.

Helena Zanella Prates et al. 2017. Netflix e a estética do banco de dados. (2017).

Andrea Samson, Sylvia Kreibig, Blake Soderstrom, A Wade, and James Gross. 2015. Eliciting positive, negative, and mixed emotional states: A film library for affective scientists. Cognition and Emotion 30, 5 (12 2015), 827–856. https://doi.org/10.1080/02699931.2015.1031089

Rose Marie SANTINI. 2020. O algoritmo do gosto: os sistemas de recomendação on-line e seus impactos no mercado cultural. vol. 1. Editora Appris, Av. Manoel Ribas, 2265 - Mercês Curitiba, Paraná, Brasil.

Rene Santos, Jorge Oliveira, Jessica Rocha, and Janaina Giraldi. 2015. Eye Tracking in Neuromarketing: A Research Agenda for Marketing Studies. International Journal of Psychological Studies 7 (02 2015). https://doi.org/10.5539/ijps.v7n1p32

Alexandre Schaefer, Frédéric Nils, Xavier Sanchez, and Pierre Philippot. 2010. Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers. Cognition & Emotion - COGNITION EMOTION 24 (11 2010), 1153–1172. https://doi.org/10.1080/02699930903274322

Helen Sharp, Jennifer Preece, and Yvonne Rogers. 2019. Interaction Design: Beyond Human-Computer Interaction, 5th Edition. John Wiley & Sons, Inc., New York, NY, USA.

Desney Tan and Anton Nijholt. 2010. Brain-Computer Interfaces: Applying Our Minds to Human-Computer Interaction. https://doi.org/10.1007/978-1-84996-272-8

Rafael Toscano, Helder Mendonça de Souza, Sandro Filho, Jaqueline Noleto, and Valdecir Becker. 2019. HCI Methods and Practices for Audiovisual Systems and Their Potential Contribution to Universal Design for Learning: A Systematic Literature Review. 526–541. https://doi.org/10.1007/978-3-030-23560-4_38

SIMPLY USER. 2013. The comparison of accuracy and precision of eye tracking: GazeFlow vs. SMI RED 250. Technical Report. Kraków, Poland.

Jan van Erp, Fabien Lotte, and Michael Tangermann. 2012. Brain-Computer Interfaces: Beyond Medical Applications. Computer 45, 4 (2012), 26–34. https://doi.org/10.1109/MC.2012.107

Jonathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert Pfurtscheller, and Theresa M. Vaughan. 2002. Brain–computer interfaces for communication and control. Clinical Neurophysiology 113, 6 (2002), 767–791. https://doi.org/10.1016/S1388-2457(02)00057-3

Songhua Xu, Hao Jiang, and Francis C.M. Lau. 2008. Personalized Online Document, Image and Video Recommendation via Commodity Eye-Tracking (Rec-Sys ’08). Association for Computing Machinery, New York, NY, USA, 83–90. https://doi.org/10.1145/1454008.1454023
Publicado
07/11/2022
Como Citar

Selecione um Formato
SILVA, Thiago Henrique Coelho Tavares Da; CAVALCANTI, Matheus Dantas; SÁ, Felipe Melo Feliciano De; MARINHO, Isaac Nóbrega; CAVALCANTI, Daniel De Queiroz; BECKER, Valdecir. Visualization of brainwaves using EEG to map emotions with eye tracking to identify attention in audiovisual workpieces. In: SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 28. , 2022, Curitiba. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2022 . p. 409-417.