A proposal of a standardized questionnaire for evaluating the User Experience in immersive Virtual Reality environments

• Jean F. P. Cheiran Federal University of Pampa / Federal University of Rio Grande do Sul http://orcid.org/0000-0002-9466-6908
• Denise R. Bandeira Federal University of Rio Grande do Sul https://orcid.org/0000-0001-9867-2718
• Marcelo S. Pimenta Federal University of Rio Grande do Sul https://orcid.org/0000-0002-4712-089X

DOI: https://doi.org/10.5753/ihc_estendido.2022.225622

Abstract

As Virtual Reality (VR) applications get popular, the need for quick, inexpensive, and reliable User Experience (UX) evaluation methods also increases. However, the lack of standardized questionnaires for assessing the UX in VR leads researchers and practitioners to use less accurate evaluation strategies. Hence, this study aims to develop a standardized questionnaire in Brazilian Portuguese for assessing the UX in immersive VR and to evaluate the questionnaire's psychometric properties. Therefore, we establish a theoretical model, create a questionnaire based on content analysis and experts' feedback, and evaluate the suitability of the measurement model (the questionnaire) and the structural model (the theoretical model).

Keywords: user experience, UX, virtual reality, evaluation, questionnaire, standardized questionnaire

References

AERA, APA, and NCME (2014). Standards for Educational and Psychological Testing. American Educational Research Association, Washington, DC, EUA.

Barbosa, S. D. J., Silva, B. S. d., Silveira, M. S., Gasparini, I., Darin, T., and Barbosa, G. D. J. (2021). Interaçãao Humano-Computador e Experiência do Usuário. autopublicação, s.l.

Blythe, M., Hassenzahl, M., Law, E., and Vermeeren, A. (2007). An analysis framework for user experience (ux) studies: a green paper. In COST294-MAUSE affiliated workshop, pages 1–5, Lancaster, RU. COST294-MAUSE.

Chang, E., Kim, H. T., and Yoo, B. (2020). Virtual Reality Sickness: A Review of Causes and Measurements. International Journal of Human-Computer Interaction, 36(17):1658–1682.

Cheng, L. K., Chieng, M. H., and Chieng, W. H. (2014). Measuring virtual experience in a three-dimensional virtual reality interactive simulator environment: A structural equation modeling approach. Virtual Reality, 18(3):173–188.

Chertoff, D. B., Goldiez, B., and LaViola, J. J. (2010). Virtual experience test: A virtual environment evaluation questionnaire. In IEEE Virtual Reality Conference, Waltham, MA, EUA. IEEE.

Cho, E. (2016). Making Reliability Reliable: A Systematic Approach to Reliability Coefficients. Organizational Research Methods, 19(4):651–682.

de Carvalho, M. R., da Costa, R. T., and Nardi, A. E. (2011). Simulator Sickness Questionnaire: tradução e adaptação transcultural. Jornal Brasileiro de Psiquiatria, 60(4):0–5.

Grassini, S. and Laumann, K. (2020). Questionnaire measures and physiological correlates of presence: A systematic review. Frontiers in Psychology, 11:349.

Hassenzahl, M., Burmester, M., and Koller, F. (2003). AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In Mensch & Computer 2003, pages 187–196. Vieweg+Teubner Verlag, Wiesbaden, Alemanha.

Hornbaek, K. and Law, E. L.-C. (2007). Meta-analysis of correlations among usability measures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, page 617–626, New York, NY, EUA. ACM.

Hutz, C. S., Bandeira, D. R., and Trentini, C. M., editors (2015). Psicometria. Artmed, Porto Alegre, RS, Brasil.

ISO (2019). ISO 9241-210:2019 - ergonomics of human-system interaction - part 210: Human-centred design for interactive systems. Standard, International Organization for Standardization, Genebra, Suíça.

Kim, Y. M., Rhiu, I., and Yun, M. H. (2020). A Systematic Review of a Virtual Reality System from the Perspective of User Experience. International Journal of Human-Computer Interaction, 36(10):893–910.

Knekta, E., Runyon, C., and Eddy, S. (2018). One size doesn’t fit all: Using factor analysis to gather validity evidence when using surveys in your research. CBE—Life Sciences Education, 18(1):1–17.

Law, E. L. (2011). The measurability and predictability of user experience. In Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems, page 1–10, New York, NY, EUA. ACM.

Mahlke, S. (2008). User Experience of Interaction with Technical Systems. Doktor der ingenieurwissenschaften, Technische Universität Berlin, Fakultät V, Alemanha.

Martins, A. I., Rosa, A. F., Queirós, A., Silva, A., and Rocha, N. P. (2015). European Portuguese validation of the System Usability Scale (SUS). Procedia - Procedia Computer Science, 67:293–300.

Melo, B. and Darin, T. (2019). Scope and definition of user experience in brazil: A survey to explore community's perspectives. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems, pages 1–11, New York, NY, EUA. ACM.

Minge, M., Thüring, M., Wagner, I., and Kuhr, C. V. (2017). The mecue questionnaire: A modular tool for measuring user experience. In Advances in Ergonomics Modeling, Usability & Special Populations, pages 115–128, Cham, Suíça. Springer.

Moraes, R. (1999). Análise de conteúdo. Revista Educação, 22(37):7–32.

Pettersson, I., Lachner, F., Frison, A.-K., Riener, A., and Butz, A. (2018). A Bermuda Triangle? - A Review of Method Application and Triangulation in User Experience Evaluation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–16, Montréal, QC, Canadá. ACM.

Preston, C. C. and Colman, A. M. (2000). Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1):1–15.

Rivero, L. and Conte, T. (2017). A systematic mapping study on research contributions on ux evaluation technologies. In Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems, pages 1–10, New York, NY, EUA. ACM.

Sampieri, R. H., Collado, C. F., and Lucio, M. P. B. (2013). Metodologia de pesquisa. Grupo A, Porto Alegre, RS, Brasil.

Sauro, J. and Lewis, J. (2016). Quantifying the User Experience: Practical Statistics for User Research. Elsevier, Cambridge, MA, EUA, 2nd edition.

Schrepp, M., Hinderks, A., and Thomaschewski, J. (2017). Construction of a Benchmark for the User Experience Questionnaire (UEQ). International Journal of Interactive Multimedia and Artificial Intelligence, 4(4):40–44.

Schrepp, M. and Thomaschewski, J. (2019). Design and Validation of a Framework for the Creation of User Experience Questionnaires. International Journal of Interactive Multimedia and Artificial Intelligence, 5(7):88–95.

Silva, G. R., Donat, J. C., Rigoli, M. M., de Oliveira, F. R., and Kristensen, C. H. (2016). A questionnaire for measuring presence in virtual environments: factor analysis of the presence questionnaire and adaptation into Brazilian Portuguese. Virtual Reality, 20(4):237–242.

Tcha-Tokey, K., Christmann, O., Loup-Escande, E., and Richir, S. (2016). Proposition and Validation of a Questionnaire to Measure the User Experience in Immersive Virtual Environments. International Journal of Virtual Reality, 16(1):33–48.

Tcha-Tokey, K., Loup-Escande, E., Christmann, O., Canac, G., Farin, F., and Richir, S. (2018). Towards a user experience model in immersive virtual environments. Advances in Human-Computer Interaction, 2018:1–10.

Vermeeren, A., Law, E., Roto, V., Obrist, M., Hoonhout, J., and Väänänen-Vainio-Mattila, K. (2010). User experience evaluation methods: Current state and development needs. In Proceedings of the Nordic Conference on Human-Computer Interaction, pages 521–530, New York, NY, EUA. ACM.