Exploring user feedback in VR: the added value of qualitative evaluation methods
Abstract
Evaluating immersive experiences, such as Virtual Reality (VR), is often done through User Experience (UX) and Quality of Experience (QoE) evaluations. Here, methodologies such as conducting experiments, the use of questionnaires or data logging can provide valuable insights into implicit user behaviour in their interactions with interactive media experiences (IMX). These quantitative methods can be combined with qualitative methods for user evaluation purposes, making it a mixed methods approach. Consulting literature, extensive research has been done to discover the various benefits and pitfalls of incorporating a mixed methods approach in evaluation studies. However, there is limited research on the added value of a qualitative research methodology in VR user evaluation studies and how these qualitative findings can provide novel insights. This paper presents a mixed-method user evaluation study of VR Planica, a novel VR experience developed by HSLU together with AFP, RTVSLO, Sparknews and VUB as part of the EU project TRANSMIXR. We begin by outlining the conceptual differences between QoE and UX, as well as the conceptual outline of mixed method approaches for user evaluation. Next, we describe our evaluation set-up, and our mixed methods approach applied in the VR Planica user evaluation study involving 33 participants. Drawing on the qualitative data, we reveal insights into how participants experienced and interpreted the experience, which helped deepen the interpretation of the quantitative results. We conclude by emphasizing the added value of a qualitative methodology in VR user evaluation studies and advocate for the use of mixed method approaches to better capture user perspectives.
References
Christine Bauer and Alexander Novotny. 2017. A consolidated view of context for intelligent systems. Journal of Ambient Intelligence and Smart Environments 9, 4 (2017), 377–393. DOI: 10.3233/AIS-170445
Edgar Bañuelos-Lozoya, Gabriel González-Serna, Nimrod González-Franco, Olivia Fragoso-Diaz, and Noé Castro-Sánchez. 2021. A Systematic Review for Cognitive State-Based QoE/UX Evaluation. Sensors 21, 10 (May 2021), 3439. DOI: 10.3390/s21103439
Artur Becker and Carla M. Dal Sasso Freitas. 2024. Evaluation of XR Applications: A Tertiary Review. Comput. Surveys 56, 5 (May 2024), 1–35. DOI: 10.1145/3626517
Julia Brannen. 2005. Mixing Methods: The Entry of Qualitative and Quantitative Approaches into the Research Process. International Journal of Social Research Methodology 8, 3 (July 2005), 173–184. DOI: 10.1080/13645570500154642
Alan Bryman. 2012. Social research methods (4. ed ed.). Oxford Univ. Press, Oxford.
John W. Creswell. 2009. Research design: qualitative, quantitative, and mixed methods approaches (3rd ed ed.). Sage Publications, Thousand Oaks, Calif. OCLC: ocn192045753.
K. Hammarberg, M. Kirkman, and S. De Lacey. 2016. Qualitative research methods: when to use them and how to judge them. Human Reproduction 31, 3 (March 2016), 498–501. DOI: 10.1093/humrep/dev334
Bingjun Han, Xin Zhang, Yifei Qi, Yuehong Gao, and Dacheng Yang. 2012. QoE Model Based Optimization for Streaming Media Service Considering Equipment and Environment Factors. Wireless Personal Communications 66, 3 (Oct. 2012), 595–612. DOI: 10.1007/s11277-012-0739-7
Philipp Haslbauer, Mike Pullen, Matej Praprotnik, Sabrina Povsic Stimec, Marie Hospital, Alexandre Grosbois, Carolin Reichherzer, and Aljosa Smolic. 2025. VR Planica: Gaussian Splatting Workflows for Immersive Storytelling. In Proceedings of ACM IMX Workshops, June 3–6, 2025, Rio de Janeiro, Brazil. SBC, Porto Alegre/RS, Brazil, 7. DOI: 10.5753/imxw.2025.8141
Dragos,-Daniel Iordache. 2024. Approaches and Theoretical Models in the Evaluation of XR Technologies Acceptance. Romanian Cyber Security Journal 6, 2 (Dec. 2024), 95–106. DOI: 10.54851/v6i2y202409
M.G. Jervis and M.A. Drake. 2014. The Use of Qualitative Research Methods in Quantitative Science: A Review. Journal of Sensory Studies 29, 4 (Aug. 2014), 234–247. DOI: 10.1111/joss.12101
Satu Jumisko-Pyykkö and Teija Vainio. 2010. Framing the Context of Use for Mobile HCI. International Journal of Mobile Human Computer Interaction 2, 4 (2010), 1–28. DOI: 10.4018/jmhci.2010100101
Zuheir N. Khlaif, Allam Mousa, and Mageswaran Sanmugam. 2024. Immersive Extended Reality (XR) Technology in Engineering Education: Opportunities and Challenges. Technology, Knowledge and Learning 29, 2 (June 2024), 803–826. DOI: 10.1007/s10758-023-09719-w
Panagiotis Kourtesis. 2024. A Comprehensive Review of Multimodal XR Applications, Risks, and Ethical Challenges in the Metaverse. Multimodal Technologies and Interaction 8, 11 (Nov. 2024), 98. DOI: 10.3390/mti8110098
Piotr Krawczyk, Marcin Topolewski, and Marc Pallot. 2017. Towards a reliable and valid mixed methods instrument in user eXperience studies. In 2017 International Conference on Engineering, Technology and Innovation (ICE/ITMC). IEEE, Funchal, 1455–1464. DOI: 10.1109/ICE.2017.8280054
Carine Lallemand and Vincent Koenig. 2020. Measuring the Contextual Dimension of User Experience: Development of the User Experience Context Scale (UXCS). In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. ACM, Tallinn Estonia, 1–13. DOI: 10.1145/3419249.3420156
Hyunkook Lee. 2020. A Conceptual Model of Immersive Experience in Extended Reality. DOI: 10.31234/osf.io/sefkh
Alexander Raake and Sebastian Egger. 2014. Quality and Quality of Experience. In Quality of Experience, Sebastian Möller and Alexander Raake (Eds.). Springer International Publishing, Cham, 11–33. DOI: 10.1007/978-3-319-02681-7_2, Series Title: T-Labs Series in Telecommunication Services.
Eeva Raita and Antti Oulasvirta. 2011. Too good to be bad: Favorable product expectations boost subjective usability ratings. Interacting with Computers 23, 4 (July 2011), 363–371. DOI: 10.1016/j.intcom.2011.04.002
J. Ruan and D. Xie. 2021. A Survey on QoE-Oriented VR Video Streaming: Some Research Issues and Challenges. Electronics 10, 17 (2021), 2155.
Camille Sagnier, Emilie Loup-Escande, Domitile Lourdeaux, Indira Thouvenin, and Gérard Vallery. 2020. User Acceptance of Virtual Reality: An Extended Technology Acceptance Model. International Journal of Human-Computer Interaction 36, 11 (Jan. 2020), 993–1007. DOI: 10.1080/10447318.2019.1708612
Ina Wechsung and Katrien De Moor. 2014. Quality of Experience Versus User Experience. In Quality of Experience: Advanced Concepts, Applications and Methods, Sebastian Möller and Alexander Raake (Eds.). Springer International Publishing, Cham, 35–54. DOI: 10.1007/978-3-319-02681-7_3
Wanmin Wu, Ahsan Arefin, Raoul Rivas, Klara Nahrstedt, Renata Sheppard, and Zhenyu Yang. 2009. Quality of experience in distributed interactive multimedia environments: toward a theoretical framework. In Proceedings of the 17th ACM international conference on Multimedia. ACM, Beijing China, 481–490. DOI: 10.1145/1631272.1631338
Tatsuya Yamazaki. 2021. Quality of Experience (QoE) Studies: Present State and Future Prospect. IEICE Transactions on Communications E104.B, 7 (July 2021), 716–724. DOI: 10.1587/transcom.2020CQI0003
