Ongoing Challenges of Evaluating Mulsemedia QoE
Abstract
Multimedia applications are usually limited to stimulating only two human senses: vision and hearing. Recent studies seek to expand the definition of multimedia applications to include stimuli for other human senses. In this way, sensory effects that should be triggered in synchrony with the audiovisual content being presented are included in the applications. By including sensory effects in multimedia, we aim to improve the Quality of Experience (QoE) with these mulsemedia environments. Usually, two approaches are being used for performing QoE evaluations these environments. The first, more common, is performed by subjective evaluation approaches, i.e. through questionnaires, interrogations, oral responses, etc. The second, rarer but growing, uses objective approaches by collecting physiological data from the user when dealing with the system being evaluated. Such data is gathered in real time or not, however, it is considered objective because it is "involuntary", that is, data is not the result of the user’s intention. This paper will address both the these methods to evaluate QoE and what the respective obstacles are when dealing with in mulsemedia systems.
Keywords:
Mulsemedia, Quality of Experience, Objective Evaluations, Subjective Evaluations
References
Ebrahim Babaei, Benjamin Tag, Tilman Dingler, and Eduardo Velloso. 2021. A Critique of Electrodermal Activity Practices at CHI. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445370
Mathias Benedek and Christian Kaernbach. 2010. A continuous measure of phasic electrodermal activity. Journal of Neuroscience Methods 190, 1 (2010), 80–91. https://doi.org/10.1016/j.jneumeth.2010.04.028
Estêvão B. Saleme, Celso A. S. Santos, Ricardo A. Falbo, Gheorghita Ghinea, and Frederic Andres. 2019. MulseOnto: a Reference Ontology to Support the Design of Mulsemedia Systems. JUCS - Journal of Universal Computer Science 25, 13 (2019), 1761–1786. https://doi.org/10.3217/jucs-025-13-1761 arXiv: https://doi.org/10.3217/jucs-025-13-1761
Gemma A. Calvert and Thomas Thesen. 2004. Multisensory integration: methodological approaches and emerging principles in the human brain. Journal of Physiology-Paris 98, 1 (2004), 191–205. https://doi.org/10.1016/j.jphysparis.2004.03.018 Representation of 3-D Space Using Different Senses In Different Species.
Thibault Chabin, Damien Gabriel, Emmanuel Haffen, Thierry Moulin, and Lionel Pazart. 2021. Are the new mobile wireless EEG headsets reliable for the evaluation of musical pleasure? PLOS ONE 15, 12 (12 2021), 1–19. https://doi.org/10.1371/journal.pone.0244820
Monica Ionita Ciolacu and Paul Svasta. 2021. Education 4.0: AI Empowers Smart Blended Learning Process with Biofeedback. In 2021 IEEE Global Engineering Education Conference (EDUCON). 1443–1448. https://doi.org/10.1109/EDUCON46332.2021.9453959
Alexandra Covaci, Ramona Trestian, Estêvão Bissoli Saleme, Ioan-Sorin Comsa,Gebremariam Assres, Celso A. S. Santos, and Gheorghita Ghinea. 2019. 360º Mulsemedia: A Way to Improve Subjective QoE in 360º Videos. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). Association for Computing Machinery, New York, NY, USA, 2378–2386. https://doi.org/10.1145/3343031.3350954
Didar Dadebayev, Wei Wei Goh, and Ee Xion Tan. 2021. EEG-based emotion recognition: Review of commercial EEG devices and machine learning techniques. Journal of King Saud University - Computer and Information Sciences (2021). https://doi.org/10.1016/j.jksuci.2021.03.009
Yi Ding, Yaqin Cao, Qingxing Qu, and Vincent G. Duffy. 2020. An Exploratory Study Using Electroencephalography (EEG) to Measure the Smartphone User Experience in the Short Term. International Journal of Human–Computer Interaction36,11(2020),1008–1021. https://doi.org/10.1080/10447318.2019.1709330
Aviv Elor, Michael Powell, Evanjelin Mahmoodi, Nico Hawthorne, Mircea Teodorescu, and Sri Kurniawan. 2020. On Shooting Stars: Comparing CAVE and HMD Immersive Virtual Reality Exergaming for Adults with Mixed Ability. ACM Trans. Comput. Healthcare 1, 4, Article 22 (sep 2020), 22 pages. https://doi.org/10.1145/3396249
Jérémy Frey, Gilad Ostrin, May Grabli, and Jessica R. Cauchard. 2020. Physiologically Driven Storytelling: Concept and Software Tool. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376643
Gheorghita Ghinea, Christian Timmerer, Weisi Lin, and Stephen R. Gulliver. 2014. Mulsemedia: State of the Art, Perspectives, and Challenges. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s, Article 17 (oct 2014), 23 pages. https://doi.org/10.1145/2617994
Giorgos Giannakakis, Dimitris Grigoriadis, Katerina Giannakaki, Olympia Simantiraki, Alexandros Roniotis, and Manolis Tsiknakis. 2019. Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing(2019),1–1. https://doi.org/10.1109/TAFFC.2019.2927337
Ai Van Thuy Ho, Karin Toska, and Jarlis Wesche. 2020. Rapid, Large, and Synchronous Sweat and Cardiovascular Responses Upon Minor Stimuli in Healthy Subjects. Dynamics and Reproducibility. Frontiers in Neurology 11 (2020). https://doi.org/10.3389/fneur.2020.00051
Samory Houzangbe, Olivier Christmann, Geoffrey Gorisse, and Simon Richir. 2020. Effects of voluntary heart rate control on user engagement and agency in a virtual reality game. Virtual Reality 24, 4 (01 Dec 2020), 665–681. https://doi.org/10.1007/s10055-020-00429-7
Sinh Huynh, Seungmin Kim, JeongGil Ko, Rajesh Krishna Balan, and Youngki Lee. 2018. EngageMon: Multi-Modal Engagement Sensing for Mobile Games. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1, Article 13 (mar 2018), 27 pages. https://doi.org/10.1145/3191745
Tao Lin, Masaki Omata, Wanhua Hu, and Atsumi Imamiya. 2005. Do Physiological Data Relate to Traditional Usability Indexes?. In Proceedings of the 17th Australia Conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future (Canberra, Australia) (OZCHI ’05). Computer-Human Interaction Special Interest Group (CHISIG) of Australia, Narrabundah, AUS, 1–10.
He Liu, Yadong Wang, and Lei Wang. 2014. The Effect of Light Conditions on Photoplethysmographic Image Acquisition Using a Commercial Camera. IEEE Journal of Translational Engineering in Health and Medicine 2 (2014), 1–11. https://doi.org/10.1109/JTEHM.2014.2360200
Ryo Maie and Robert M. DeKeyser. 2020. CONFLICTING EVIDENCE OF EXPLICIT AND IMPLICIT KNOWLEDGE FROM OBJECTIVE AND SUBJECTIVE MEASURES. Studies in Second Language Acquisition 42, 2 (2020), 359–382. https://doi.org/10.1017/S0272263119000615
M. G McKee. 2008. Biofeedback: an overview in the context of heart-brain medicine. Cleveland Clinic Journal of Medicine 75, Suppl_2 (March 2008), S31–S31. https://doi.org/10.3949/ccjm.75.suppl_2.s31
Niall Murray, Oluwakemi A. Ademoye, Gheorghita Ghinea, and Gabriel-Miro Muntean. 2017. A Tutorial for Olfaction-Based Multisensorial Media Application Design and Evaluation. ACM Comput. Surv. 50, 5, Article 67 (sep 2017), 30 pages. https://doi.org/10.1145/3108243
Jodi Oakman, Margo Ketels, and Els Clays. 2021. Low back and neck pain: objective and subjective measures of workplace psychosocial and physical hazards. International Archives of Occupational and Environmental Health 94, 7 (01 Oct 2021), 1637–1644. https://doi.org/10.1007/s00420-021-01707-w
Susanne Putze, Dmitry Alexandrovsky, Felix Putze, Sebastian Höffner, Jan David Smeddinck, and Rainer Malaka. 2020. Breaking The Experience: Effects of Questionnaires in VR User Studies. Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3313831.3376144
Saeed Safikhani, Michael Holly, Alexander Kainz, and Johanna Pirker. 2021. The Influence of In-VR Questionnaire Design on the User Experience. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (Osaka, Japan) (VRST ’21). Association for Computing Machinery, New York, NY, USA, Article 12, 8 pages. https://doi.org/10.1145/3489849.3489884
E. B. Saleme, Alexandra Covaci, Gebremariam Assres, Ioan-Sorin Comsa, Ramona Trestian, Celso A.S. Santos, and Gheorghita Ghinea. 2021. The influence of human factors on 360 mulsemedia QoE. International Journal of Human-Computer Studies 146 (2021), 102550. https://doi.org/10.1016/j.ijhcs.2020.102550
Estêvão Bissoli Saleme, Celso A. S. Santos, and Gheorghita Ghinea. 2019. Coping With the Challenges of Delivering Multiple Sensorial Media. IEEE MultiMedia 26, 2 (2019), 66–75. https://doi.org/10.1109/MMUL.2018.2873565
Valentin Schwind, Pascal Knierim, Nico Haas, and Niels Henze. 2019. Using Presence Questionnaires in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300590
Mahsa Soufineyestani, Dale Dowling, and Arshia Khan. 2020. Electroencephalography (EEG) Technology Applications and Available Devices. Applied Sciences 10, 21 (2020). https://doi.org/10.3390/app10217453
Yingding Wang, Nikolai Fischer, and François Bry. 2019. Pervasive Persuasion for Stress Self-Regulation. In 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). 724–730. https://doi.org/10.1109/PERCOMW.2019.8730850
Feng Xia, Laurence T. Yang, Lizhe Wang, and Alexey Vinel. 2012. Internet of Things. Int. J. Commun. Syst. 25, 9 (sep 2012), 1101–1102. https://doi.org/10.1002/dac.2417
Xu Yan, Li-Ming Zhao, and Bao-Liang Lu. 2021. Simplifying Multimodal Emotion Recognition with Single Eye Movement Modality. Association for Computing Machinery, New York, NY, USA, 1057–1063. https://doi.org/10.1145/3474085.3475701
Xiaozhen Ye, Huansheng Ning, Per Backlund, and Jianguo Ding. 2021. Flow Experience Detection and Analysis for Game Users by Wearable-Devices-Based Physiological Responses Capture. IEEE Internet of Things Journal 8, 3 (2021), 1373–1387. https://doi.org/10.1109/JIOT.2020.3010853
Zhenhui Yuan, Ting Bi, Gabriel-Miro Muntean, and Gheorghita Ghinea. 2015. Perceived Synchronization of Mulsemedia Services. IEEE Transactions on Multimedia 17,7 (2015), 957–966. https://doi.org/10.1109/TMM.2015.2431915
Mathias Benedek and Christian Kaernbach. 2010. A continuous measure of phasic electrodermal activity. Journal of Neuroscience Methods 190, 1 (2010), 80–91. https://doi.org/10.1016/j.jneumeth.2010.04.028
Estêvão B. Saleme, Celso A. S. Santos, Ricardo A. Falbo, Gheorghita Ghinea, and Frederic Andres. 2019. MulseOnto: a Reference Ontology to Support the Design of Mulsemedia Systems. JUCS - Journal of Universal Computer Science 25, 13 (2019), 1761–1786. https://doi.org/10.3217/jucs-025-13-1761 arXiv: https://doi.org/10.3217/jucs-025-13-1761
Gemma A. Calvert and Thomas Thesen. 2004. Multisensory integration: methodological approaches and emerging principles in the human brain. Journal of Physiology-Paris 98, 1 (2004), 191–205. https://doi.org/10.1016/j.jphysparis.2004.03.018 Representation of 3-D Space Using Different Senses In Different Species.
Thibault Chabin, Damien Gabriel, Emmanuel Haffen, Thierry Moulin, and Lionel Pazart. 2021. Are the new mobile wireless EEG headsets reliable for the evaluation of musical pleasure? PLOS ONE 15, 12 (12 2021), 1–19. https://doi.org/10.1371/journal.pone.0244820
Monica Ionita Ciolacu and Paul Svasta. 2021. Education 4.0: AI Empowers Smart Blended Learning Process with Biofeedback. In 2021 IEEE Global Engineering Education Conference (EDUCON). 1443–1448. https://doi.org/10.1109/EDUCON46332.2021.9453959
Alexandra Covaci, Ramona Trestian, Estêvão Bissoli Saleme, Ioan-Sorin Comsa,Gebremariam Assres, Celso A. S. Santos, and Gheorghita Ghinea. 2019. 360º Mulsemedia: A Way to Improve Subjective QoE in 360º Videos. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). Association for Computing Machinery, New York, NY, USA, 2378–2386. https://doi.org/10.1145/3343031.3350954
Didar Dadebayev, Wei Wei Goh, and Ee Xion Tan. 2021. EEG-based emotion recognition: Review of commercial EEG devices and machine learning techniques. Journal of King Saud University - Computer and Information Sciences (2021). https://doi.org/10.1016/j.jksuci.2021.03.009
Yi Ding, Yaqin Cao, Qingxing Qu, and Vincent G. Duffy. 2020. An Exploratory Study Using Electroencephalography (EEG) to Measure the Smartphone User Experience in the Short Term. International Journal of Human–Computer Interaction36,11(2020),1008–1021. https://doi.org/10.1080/10447318.2019.1709330
Aviv Elor, Michael Powell, Evanjelin Mahmoodi, Nico Hawthorne, Mircea Teodorescu, and Sri Kurniawan. 2020. On Shooting Stars: Comparing CAVE and HMD Immersive Virtual Reality Exergaming for Adults with Mixed Ability. ACM Trans. Comput. Healthcare 1, 4, Article 22 (sep 2020), 22 pages. https://doi.org/10.1145/3396249
Jérémy Frey, Gilad Ostrin, May Grabli, and Jessica R. Cauchard. 2020. Physiologically Driven Storytelling: Concept and Software Tool. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376643
Gheorghita Ghinea, Christian Timmerer, Weisi Lin, and Stephen R. Gulliver. 2014. Mulsemedia: State of the Art, Perspectives, and Challenges. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s, Article 17 (oct 2014), 23 pages. https://doi.org/10.1145/2617994
Giorgos Giannakakis, Dimitris Grigoriadis, Katerina Giannakaki, Olympia Simantiraki, Alexandros Roniotis, and Manolis Tsiknakis. 2019. Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing(2019),1–1. https://doi.org/10.1109/TAFFC.2019.2927337
Ai Van Thuy Ho, Karin Toska, and Jarlis Wesche. 2020. Rapid, Large, and Synchronous Sweat and Cardiovascular Responses Upon Minor Stimuli in Healthy Subjects. Dynamics and Reproducibility. Frontiers in Neurology 11 (2020). https://doi.org/10.3389/fneur.2020.00051
Samory Houzangbe, Olivier Christmann, Geoffrey Gorisse, and Simon Richir. 2020. Effects of voluntary heart rate control on user engagement and agency in a virtual reality game. Virtual Reality 24, 4 (01 Dec 2020), 665–681. https://doi.org/10.1007/s10055-020-00429-7
Sinh Huynh, Seungmin Kim, JeongGil Ko, Rajesh Krishna Balan, and Youngki Lee. 2018. EngageMon: Multi-Modal Engagement Sensing for Mobile Games. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1, Article 13 (mar 2018), 27 pages. https://doi.org/10.1145/3191745
Tao Lin, Masaki Omata, Wanhua Hu, and Atsumi Imamiya. 2005. Do Physiological Data Relate to Traditional Usability Indexes?. In Proceedings of the 17th Australia Conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future (Canberra, Australia) (OZCHI ’05). Computer-Human Interaction Special Interest Group (CHISIG) of Australia, Narrabundah, AUS, 1–10.
He Liu, Yadong Wang, and Lei Wang. 2014. The Effect of Light Conditions on Photoplethysmographic Image Acquisition Using a Commercial Camera. IEEE Journal of Translational Engineering in Health and Medicine 2 (2014), 1–11. https://doi.org/10.1109/JTEHM.2014.2360200
Ryo Maie and Robert M. DeKeyser. 2020. CONFLICTING EVIDENCE OF EXPLICIT AND IMPLICIT KNOWLEDGE FROM OBJECTIVE AND SUBJECTIVE MEASURES. Studies in Second Language Acquisition 42, 2 (2020), 359–382. https://doi.org/10.1017/S0272263119000615
M. G McKee. 2008. Biofeedback: an overview in the context of heart-brain medicine. Cleveland Clinic Journal of Medicine 75, Suppl_2 (March 2008), S31–S31. https://doi.org/10.3949/ccjm.75.suppl_2.s31
Niall Murray, Oluwakemi A. Ademoye, Gheorghita Ghinea, and Gabriel-Miro Muntean. 2017. A Tutorial for Olfaction-Based Multisensorial Media Application Design and Evaluation. ACM Comput. Surv. 50, 5, Article 67 (sep 2017), 30 pages. https://doi.org/10.1145/3108243
Jodi Oakman, Margo Ketels, and Els Clays. 2021. Low back and neck pain: objective and subjective measures of workplace psychosocial and physical hazards. International Archives of Occupational and Environmental Health 94, 7 (01 Oct 2021), 1637–1644. https://doi.org/10.1007/s00420-021-01707-w
Susanne Putze, Dmitry Alexandrovsky, Felix Putze, Sebastian Höffner, Jan David Smeddinck, and Rainer Malaka. 2020. Breaking The Experience: Effects of Questionnaires in VR User Studies. Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3313831.3376144
Saeed Safikhani, Michael Holly, Alexander Kainz, and Johanna Pirker. 2021. The Influence of In-VR Questionnaire Design on the User Experience. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (Osaka, Japan) (VRST ’21). Association for Computing Machinery, New York, NY, USA, Article 12, 8 pages. https://doi.org/10.1145/3489849.3489884
E. B. Saleme, Alexandra Covaci, Gebremariam Assres, Ioan-Sorin Comsa, Ramona Trestian, Celso A.S. Santos, and Gheorghita Ghinea. 2021. The influence of human factors on 360 mulsemedia QoE. International Journal of Human-Computer Studies 146 (2021), 102550. https://doi.org/10.1016/j.ijhcs.2020.102550
Estêvão Bissoli Saleme, Celso A. S. Santos, and Gheorghita Ghinea. 2019. Coping With the Challenges of Delivering Multiple Sensorial Media. IEEE MultiMedia 26, 2 (2019), 66–75. https://doi.org/10.1109/MMUL.2018.2873565
Valentin Schwind, Pascal Knierim, Nico Haas, and Niels Henze. 2019. Using Presence Questionnaires in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300590
Mahsa Soufineyestani, Dale Dowling, and Arshia Khan. 2020. Electroencephalography (EEG) Technology Applications and Available Devices. Applied Sciences 10, 21 (2020). https://doi.org/10.3390/app10217453
Yingding Wang, Nikolai Fischer, and François Bry. 2019. Pervasive Persuasion for Stress Self-Regulation. In 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). 724–730. https://doi.org/10.1109/PERCOMW.2019.8730850
Feng Xia, Laurence T. Yang, Lizhe Wang, and Alexey Vinel. 2012. Internet of Things. Int. J. Commun. Syst. 25, 9 (sep 2012), 1101–1102. https://doi.org/10.1002/dac.2417
Xu Yan, Li-Ming Zhao, and Bao-Liang Lu. 2021. Simplifying Multimodal Emotion Recognition with Single Eye Movement Modality. Association for Computing Machinery, New York, NY, USA, 1057–1063. https://doi.org/10.1145/3474085.3475701
Xiaozhen Ye, Huansheng Ning, Per Backlund, and Jianguo Ding. 2021. Flow Experience Detection and Analysis for Game Users by Wearable-Devices-Based Physiological Responses Capture. IEEE Internet of Things Journal 8, 3 (2021), 1373–1387. https://doi.org/10.1109/JIOT.2020.3010853
Zhenhui Yuan, Ting Bi, Gabriel-Miro Muntean, and Gheorghita Ghinea. 2015. Perceived Synchronization of Mulsemedia Services. IEEE Transactions on Multimedia 17,7 (2015), 957–966. https://doi.org/10.1109/TMM.2015.2431915
Published
2022-06-22
How to Cite
SILVEIRA, Aleph Campos da; SANTOS, Celso Alberto Saibel.
Ongoing Challenges of Evaluating Mulsemedia QoE. In: WORKSHOP ON MULTISENSORY EXPERIENCES (SENSORYX), 2. , 2022, Aveiro.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
DOI: https://doi.org/10.5753/sensoryx.2022.20005.
