Guidelines for conducting biofeedback-enhanced QoE studies in mulsemedia-enhanced virtual reality

  • Aleph Silveira UFES
  • Roope Raisamo Tampere University
  • Fotios Spyridonis Brunel University London
  • Alexandra Covaci University of Kent
  • George Ghinea Brunel University London
  • Celso A. S. Santos UFES


This paper presents the conclusions drawn from an ongoing experiment investigating the performance, usability, and impact of multisensory stimuli in the virtual environment, with a focus on proposing guidelines for conducting biofeedback-enhanced Quality of Experience (QoE) studies in multimedia-enhanced Virtual Reality (VR). The study evaluated various devices, including the Polar H10 and Grove GSR for biosignal measurement and an EEG device for brainwave analysis. Concerns related to participant comfort were highlighted, such as discomfort caused by tight electrodes and difficulties in achieving consistent contact with the scalp. Ergonomic issues with Head-Mounted Displays (HMDs) were also identified, emphasizing the need for comfortable and immersive experiences. The paper recommends addressing these concerns through inclusive design and user-friendly adaptation of devices. The findings emphasize the importance of integrated devices and user-friendly design to enhance QoE and facilitate the adoption of biofeedback technologies outside of the lab. By following the proposed guidelines, researchers and developers can improve the immersive experience and advance the field of biofeedback in VR environments.

Palavras-chave: User Evaluation, Biofeedback, Guidelines, Quality of Experience, Mulsemedia


Mohamed Abdelazez, Sreeraman Rajan, and Adrian D.C. Chan. 2021. Automated Biosignal Quality Analysis of Electrocardiograms. IEEE Instrumentation & Measurement Magazine 24, 2 (2021), 37–44.

Oluwakemi A. Ademoye, Niall Murray, Gabriel-Miro Muntean, and Gheorghita Ghinea. 2016. Audio Masking Effect on Inter-Component Skews in Olfaction-Enhanced Multimedia Presentations. ACM Trans. Multimedia Comput. Commun. Appl. 12, 4, Article 51 (aug 2016), 14 pages.

Tayebeh Baniasadi, Seyed Mohammad Ayyoubzadeh, and Niloofar Mohammadzadeh. 2020. Challenges and practical considerations in applying virtual reality in medical education and treatment. Oman medical journal 35, 3 (2020), e125

Jessica Oliveira Brito, Celso A. S. Santos, Rodrigo Laiola Guimarães, and Thiago Felipe Corrêa Borges. 2019. Toward Understanding the Quality of Subtitle Synchronization to Improve the Viewer Experience. In Proceedings of the 25th Brazillian Symposium on Multimedia and the Web (Rio de Janeiro, Brazil) (WebMedia ’19). Association for Computing Machinery, New York, NY, USA, 209–216.

Gemma A. Calvert and Thomas Thesen. 2004. Multisensory integration: methodological approaches and emerging principles in the human brain. Journal of Physiology-Paris 98, 1 (2004), 191–205. Representation of 3-D Space Using Different Senses In Different Species

Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, and Baltasar Fernández-Manjón. 2022. Scoping review of bioelectrical signals uses in videogames for evaluation purposes.IEEE Access (2022)

Sandra Cano, Jonathan Soto, Laura Acosta, Victor M. Peñeñory, and Fernando Moreira. 2022. Using Brain -Computer Interface to evaluate the User eXperience in interactive systems. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization (May 2022), 1–9.

Ioan-Sorin Comşa, Estêvão Bissoli Saleme, Alexandra Covaci, Gebremariam Mesfin Assres, Ramona Trestian, Celso A. S. Santos, and Gheorghiţă Ghinea. 2020. Do I Smell Coffee? The Tale of a 360° Mulsemedia Experience. IEEE MultiMedia 27, 1 (2020), 27–36.

Alexandra Covaci, Estêvão B Saleme, Gebremariam Mesfin, Ioan-Sorin Comsa, Ramona Trestian, Celso AS Santos, and George Ghinea. 2022. Multisensory 360 videos under varying resolution levels enhance presence. IEEE Transactions on Visualization and Computer Graphics 29, 4 (2022), 2093–2101.

Alexandra Covaci, Estêvão Bissoli Saleme, Gebremariam Mesfin, Nadia Hussain, Elahe Kani-Zabihi, and Gheorghita Ghinea. 2020. How Do We Experience Crossmodal Correspondent Mulsemedia Content?IEEE Transactions on Multimedia 22, 5 (2020), 1249–1258.

Sebastião Baümberg Tavares da Silva. 2021. A Framework for Supporting Privacy in the Computation of Biosignals. Ph.D. Dissertation. Universidade de Lisboa (Portugal)

Tarak Das, Camellia Mitra, Hritwika Paul, and Sahil Banerjee. 2022. Study the Changes of Bioelectrical Skin Impedance of Human body Associated with Different Physiological Parameters. Journal homepage: www. ijrpr. com ISSN 2582 (2022), 7421

Darragh Egan, Sean Brennan, John Barrett, Yuansong Qiao, Christian Timmerer, and Niall Murray. 2016. An evaluation of Heart Rate and ElectroDermal Activity as an objective QoE evaluation method for immersive virtual reality environments. In 2016 eighth international conference on quality of multimedia experience (QoMEX). IEEE, 1–6

Jing Gao, Peng Li, Zhikui Chen, and Jianing Zhang. 2020. A Survey on Deep Learning for Multimodal Data Fusion. Neural Computation 32, 5 (05 2020), 829–864. arxiv: [link].

George Ghinea, Frederic Andres, and Stephen Gulliver. 2012. Multiple Sensorial Media Advances and Applications: New Developments in. (2012)

Giorgos Giannakakis, Dimitris Grigoriadis, Katerina Giannakaki, Olympia Simantiraki, Alexandros Roniotis, and Manolis Tsiknakis. 2019. Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing 13, 1 (2019), 440–460

Kunal Gupta, Yuewei Zhang, Tamil Selvan Gunasekaran, Prasanth Sasikumar, Nanditha Krishna, Yun Suen Pai, and Mark Billinghurst. 2023. VRdoGraphy: An Empathic VR Photography Experience. In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 1013–1014

Marc Janeras, Joan Roca, Josep A Gili, Oriol Pedraza, Gerald Magnusson, M Amparo Núñez-Andrés, and Kathryn Franklin. 2022. Using Mixed Reality for the Visualization and Dissemination of Complex 3D Models in Geosciences—Application to the Montserrat Massif (Spain). Geosciences 12, 10 (2022), 370

Dana Lahat, Tülay Adali, and Christian Jutten. 2015. Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects. Proc. IEEE 103, 9 (2015), 1449–1477.

Yiming Liu, Chun Ki Yiu, Zhao Zhao, Wooyoung Park, Rui Shi, Xingcan Huang, Yuyang Zeng, Kuan Wang, Tsz Hung Wong, Shengxin Jia, Jingkun Zhou, Zhan Gao, Ling Zhao, Kuanming Yao, Jian Li, Chuanlu Sha, Yuyu Gao, Guangyao Zhao, Ya Huang, Dengfeng Li, Qinglei Guo, Yuhang Li, and Xinge Yu. 2023. Soft, miniaturized, wireless olfactory interface for virtual reality. Nature Communications 14, 1 (May 2023).

Olibário Machado Neto and Maria da Graça Pimentel. 2013. Heuristics for the Assessment of Interfaces of Mobile Devices. In Proceedings of the 19th Brazilian Symposium on Multimedia and the Web (Salvador, Brazil) (WebMedia ’13). Association for Computing Machinery, New York, NY, USA, 93–96.

Arian Mehrfard, Javad Fotouhi, Giacomo Taylor, Tess Forster, Nassir Navab, and Bernhard Fuerst. 2019. A comparative analysis of virtual reality head-mounted display systems. arXiv preprint arXiv:1912.02913 (2019)

Mina Mikhail, Khaled El-Ayat, Rana El Kaliouby, James Coan, and John J. B. Allen. 2010. Emotion Detection Using Noisy EEG Data. In Proceedings of the 1st Augmented Human International Conference (Megève, France) (AH ’10). Association for Computing Machinery, New York, NY, USA, Article 7, 7 pages.

Niall Murray, Oluwakemi A. Ademoye, Gheorghita Ghinea, and Gabriel-Miro Muntean. 2017. A Tutorial for Olfaction-Based Multisensorial Media Application Design and Evaluation. ACM Comput. Surv. 50, 5, Article 67 (sep 2017), 30 pages.

Niall Murray, Brian Lee, Yuansong Qiao, and Gabriel-Miro Muntean. 2014. Multiple-Scent Enhanced Multimedia Synchronization. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s, Article 12 (oct 2014), 28 pages.

Diogo Kionori Cândido Nishikawa, Roberta Pereira Brandão, and Vânia Paula de Almeida Neris. 2020. Um estudo empírico sobre reações emocionais de usuários na interação com interfaces web pautadas na Gestalt. In Anais do XI Workshop sobre Aspectos da Interação Humano-Computador para a Web Social. SBC, 25–32

Pratheep Kumar Paranthaman, Nikesh Bajaj, Nicholas Solovey, and David Jennings. 2021. Comparative Evaluation of the EEG Performance Metrics and Player Ratings on the Virtual Reality Games. In 2021 IEEE Conference on Games (CoG). IEEE, 1–8

Szczepan Paszkiel and Szczepan Paszkiel. 2020. Using BCI and VR technology in neurogaming. Analysis and Classification of EEG Signals for Brain–Computer Interfaces (2020), 93–99

Jiali Qian, Daniel J McDonough, and Zan Gao. 2020. The effectiveness of virtual reality exercise on individual’s physiological, psychological and rehabilitative outcomes: a systematic review. International journal of environmental research and public health 17, 11 (2020), 4133

Plínio M.S. Ramos, Caio B.S. Maior, Márcio C. Moura, and Isis D. Lins. 2022. Automatic drowsiness detection for safety-critical operations using ensemble models and EEG signals. Process Safety and Environmental Protection 164 (2022), 566–581.

Giuseppe Riva, Brenda K Wiederhold, and Fabrizia Mantovani. 2019. Neuroscience of virtual reality: from virtual exposure to embodied medicine. Cyberpsychology, behavior, and social networking 22, 1 (2019), 82–96

Theerat Saichoo, Poonpong Boonbrahm, and Yunyong Punsawad. 2021. Facial-Machine interface-based virtual Reality Wheelchair control using EEG artifacts of Emotiv neuroheadset. In 2021 18th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON). IEEE, 781–784

Estêvão B. Saleme, Alexandra Covaci, Gebremariam Mesfin, Celso A. S. Santos, and Gheorghita Ghinea. 2019. Mulsemedia DIY: A Survey of Devices and a Tutorial for Building Your Own Mulsemedia Environment. ACM Comput. Surv. 52, 3, Article 58 (jun 2019), 29 pages.

Débora Pereira Salgado, Felipe Roque Martins, Thiago Braga Rodrigues, Conor Keighrey, Ronan Flynn, Eduardo Lázaro Martins Naves, and Niall Murray. 2018. A QoE assessment method based on EDA, heart rate and EEG of a virtual reality assistive technology system. In Proceedings of the 9th ACM Multimedia Systems Conference. 517–520.

Marcelle Schaffarczyk, Bruce Rogers, Rüdiger Reer, and Thomas Gronwald. 2022. Validity of the polar H10 sensor for heart rate variability analysis during resting state and incremental exercise in recreational men and women. Sensors 22, 17 (2022), 6536

Catriona L. Scrivener and Arran T. Reader. 2022. Variability of EEG electrode positions and their underlying brain regions: visualizing gel artifacts from a simultaneous EEG-fMRI dataset. Brain and Behavior 12, 2 (Jan. 2022).

Jungryul Seo, Teemu H Laine, and Kyung-Ah Sohn. 2019. An exploration of machine learning methods for robust boredom classification using EEG and GSR data. Sensors 19, 20 (2019), 4561

Tanuja Subba and Tejbanta Singh Chingtham. 2022. A Review on Types of Machine Learning Techniques for Biosignal Evaluation for Human Computer Interaction. Advanced Computational Paradigms and Hybrid Intelligent Computing: Proceedings of ICACCP 2021 (2022), 457–466

Moritz Tacke, Katharina Janson, Katharina Vill, Florian Heinen, Lucia Gerstl, Karl Reiter, and Ingo Borggraefe. 2022. Effects of a reduction of the number of electrodes in the EEG montage on the number of identified seizure patterns. Scientific Reports 12, 1 (2022), 1–7

Tijana Vuletic, Alex Duffy, Laura Hay, Chris McTeague, Gerard Campbell, and Madeleine Grealy. 2019. Systematic literature review of hand gestures used in human computer interaction interfaces. International Journal of Human-Computer Studies 129 (2019), 74–94.

Efy Yosrita, Yaya Heryadi, Lili Ayu Wulandhari, and Widodo Budiharto. 2019. EEG Based Identification of Words on Exam Models with Yes-No Answers for Students with Visual Impairments. In 2019 IEEE International Conference on Engineering, Technology and Education (TALE). IEEE, 1–5.
SILVEIRA, Aleph; RAISAMO, Roope; SPYRIDONIS, Fotios; COVACI, Alexandra; GHINEA, George; SANTOS, Celso A. S.. Guidelines for conducting biofeedback-enhanced QoE studies in mulsemedia-enhanced virtual reality. In: SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 29. , 2023, Ribeirão Preto/SP. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2023 . p. 32–40.