Assessing the Usability and Presence of an Immersive Virtual Environment: Opportunities and Challenges of Combining SUS, IPQ and Nielsen’s Heuristics
Resumo
Ensuring the usability and effectiveness of immersive virtual environments requires robust and reliable measurement tools. This study examines the opportunities and challenges of combining the System Usability Scale (SUS), the Igroup Presence Questionnaire (IPQ), and Nielsen’s Usability Heuristics to assess usability and sense of presence in immersive virtual environments. Conducted under the Design Science Research paradigm, the study engaged undergraduate engineering students in evaluating the Spatial.io platform as an illustrative case. Results indicated marginally acceptable usability (SUS score 67.5) and low-to-moderate presence (IPQ subscales: spatial presence 1.46, involvement 2.92, realism 1.19). Heuristic evaluation revealed strengths such as visible system status and real-world alignment, but also limitations including restricted user control and limited error prevention. The study concludes that combining heuristic and standardized instruments provides a more comprehensive framework for assessing immersive systems, integrating quantitative usability and presence data with qualitative interaction insights to guide design improvement.
Palavras-chave:
virtual reality, usability, presence, Nielsen heuristics, immersive virtual environments
Referências
M. Slater and M. V. Sanchez-Vives. Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3, 2016. DOI: 10.3389/frobt.2016.00074. Available [link]
J. Radianti, T. A. Majchrzak, J. Fromm, and I. Wohlgenannt. A systematic review of immersive virtual reality applications for higher education Design elements, lessons learned, and research agenda. Computers & Education, 147, 2020, art. no. 103778. DOI: 10.1016/j.compedu.2019.103778. Available [link]
L. G. G. Almeida, N. V. d. Vasconcelos, I. Winkler, and M. F. Catapan. Innovating industrial training with immersive metaverses A method for developing cross-platform virtual reality environments. Applied Sciences, 13(15), 2023. Available [link]
G. Brito, D. Scheltes, A. Durval, J. Aragao, S. Santos, C. Lima, M. Soussa, and I. Winkler. Developing an university immersive campus Guidelines for beginner developers. In Anais do XXVI Simposio de Realidade Virtual e Aumentada, Porto Alegre, RS, Brasil, SBC, 2024, 261–265. Available [link]
T. Jokela, N. Iivari, J. Matero, and M. Karukka. The standard of user-centered design and the standard definition of usability Analyzing ISO 13407 against ISO 9241-11. In Proceedings of the Latin American Conference on Human-Computer Interaction (CLIHC 03), Rio de Janeiro, Brazil, ACM, 2003. Available [link]
J. Brooke. SUS A quick and dirty usability scale. In Usability Evaluation in Industry, P. W. Jordan, B. Thomas, I. L. McClelland, and B. A. Weerdmeester, Eds., CRC Press, London, 1996, 189–194. DOI: 10.1201/9781498710411,
R. E. M. de Oliveira, L. R. Ferreira, and J. C. de Oliveira. A pedagogical virtual reality environment for children in early childhood education. In Proceedings of the 25th Symposium on Virtual and Augmented Reality (SVR 23). Association for Computing Machinery, 2024. DOI: 10.1145/3625008.3625047.
IGroup. The International Personality Item Pool (IPIP) website. Website, 2025. Available [link]. Accessed 6 June 2025.
T. Schubert, F. Friedmann, and H. Regenbrecht. The experience of presence Factor analytic insights. Presence Teleoperators and Virtual Environments, 10(3):266–281, 2001. DOI: 10.1162/105474601300343603.
J. Nielsen. Heuristic evaluation. In Usability Inspection Methods, J. Nielsen and R. L. Mack, Eds., John Wiley & Sons, New York, 1994, 25–62.
A. Sutcliffe and B. Gault. Heuristic evaluation of virtual reality applications. Interacting with Computers, 16(4):831–849, 2004. DOI: 10.1016/j.intcom.2004.05.001.
M. Roupe, M. Johansson, and P. Kreisl. Enhancing a collaborative design process using full-scale 3D virtual models. Visualization in Engineering, 4(1):1–15, 2016.
J. Nielsen. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 1994. DOI: 10.1145/191666.191729. Available [link]
J. Jerald. The VR Book Human-Centered Design for Virtual Reality. Morgan & Claypool Publishers, San Rafael, 2015. DOI: 10.1145/2792790. Available [link]
A. Mohammad and L. Pedersen. Analyzing the use of heuristics in a virtual reality learning context A literature review. Informatics, 9(3), 2022, art. no. 51. DOI: 10.3390/informatics9030051.
X. Guo, K. Nerella, J. Dong, C. Qian, and Y. Chen. Understanding pitfalls and opportunities of applying heuristic evaluation methods to VR training systems An empirical study. International Journal of Human–Computer Interaction, 40(9), 2023. DOI: 10.1080/10447318.2022.2161238.
S. Stadler, H. Cornet, and F. Frenkler. Assessing heuristic evaluation in immersive virtual reality a case study on future guidance systems. Multimodal Technologies and Interaction, 7(2):19, 2023. DOI: 10.3390/mti7020019.
S. Gregor and A. R. Hevner. Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2):337–355, 2013. DOI: 10.25300/MISQ/2013/37.2.01.
A. R. Hevner, S. T. March, J. Park, and S. Ram. Design science in information systems research. MIS Quarterly, 28(1):75–105, 2004. DOI: 10.2307/25148625.
A. Bangor, P. T. Kortum, and J. T. Miller. An empirical evaluation of the System Usability Scale. International Journal of Human–Computer Interaction, 24(6):574–594, 2008. DOI: 10.1080/10447310802205776.
A. Wright. Education in the metaverse A new way of learning. Blog post, Spatial.io, Nov. 2021. Available [link]. Accessed 26 May 2025.
Spatial. Koreatech revolutionizes vocational training with interactive virtual learning. Case study, Spatial.io, 2025. Available [link]. Accessed 26 May 2025.
R. Brown, J. Recker, and S. West. Using virtual worlds for collaborative business process modeling. Business Process Management Journal, 17(3):546–564, 2011. DOI: 10.1108/14637151111136414.
C. Schwarz and Z. Wang. The role of digital twins in connected and automated vehicles. IEEE Intelligent Transportation Systems Magazine, 14(6):41–51, 2022. DOI: 10.1109/MITS.2021.3129524.
A. Kendrick. 10 usability heuristics applied to virtual reality. Nielsen Norman Group, Jul. 2021. Available [link]. Accessed 11 Jun. 2025.
J. Radianti, T. A. Majchrzak, J. Fromm, and I. Wohlgenannt. A systematic review of immersive virtual reality applications for higher education Design elements, lessons learned, and research agenda. Computers & Education, 147, 2020, art. no. 103778. DOI: 10.1016/j.compedu.2019.103778. Available [link]
L. G. G. Almeida, N. V. d. Vasconcelos, I. Winkler, and M. F. Catapan. Innovating industrial training with immersive metaverses A method for developing cross-platform virtual reality environments. Applied Sciences, 13(15), 2023. Available [link]
G. Brito, D. Scheltes, A. Durval, J. Aragao, S. Santos, C. Lima, M. Soussa, and I. Winkler. Developing an university immersive campus Guidelines for beginner developers. In Anais do XXVI Simposio de Realidade Virtual e Aumentada, Porto Alegre, RS, Brasil, SBC, 2024, 261–265. Available [link]
T. Jokela, N. Iivari, J. Matero, and M. Karukka. The standard of user-centered design and the standard definition of usability Analyzing ISO 13407 against ISO 9241-11. In Proceedings of the Latin American Conference on Human-Computer Interaction (CLIHC 03), Rio de Janeiro, Brazil, ACM, 2003. Available [link]
J. Brooke. SUS A quick and dirty usability scale. In Usability Evaluation in Industry, P. W. Jordan, B. Thomas, I. L. McClelland, and B. A. Weerdmeester, Eds., CRC Press, London, 1996, 189–194. DOI: 10.1201/9781498710411,
R. E. M. de Oliveira, L. R. Ferreira, and J. C. de Oliveira. A pedagogical virtual reality environment for children in early childhood education. In Proceedings of the 25th Symposium on Virtual and Augmented Reality (SVR 23). Association for Computing Machinery, 2024. DOI: 10.1145/3625008.3625047.
IGroup. The International Personality Item Pool (IPIP) website. Website, 2025. Available [link]. Accessed 6 June 2025.
T. Schubert, F. Friedmann, and H. Regenbrecht. The experience of presence Factor analytic insights. Presence Teleoperators and Virtual Environments, 10(3):266–281, 2001. DOI: 10.1162/105474601300343603.
J. Nielsen. Heuristic evaluation. In Usability Inspection Methods, J. Nielsen and R. L. Mack, Eds., John Wiley & Sons, New York, 1994, 25–62.
A. Sutcliffe and B. Gault. Heuristic evaluation of virtual reality applications. Interacting with Computers, 16(4):831–849, 2004. DOI: 10.1016/j.intcom.2004.05.001.
M. Roupe, M. Johansson, and P. Kreisl. Enhancing a collaborative design process using full-scale 3D virtual models. Visualization in Engineering, 4(1):1–15, 2016.
J. Nielsen. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 1994. DOI: 10.1145/191666.191729. Available [link]
J. Jerald. The VR Book Human-Centered Design for Virtual Reality. Morgan & Claypool Publishers, San Rafael, 2015. DOI: 10.1145/2792790. Available [link]
A. Mohammad and L. Pedersen. Analyzing the use of heuristics in a virtual reality learning context A literature review. Informatics, 9(3), 2022, art. no. 51. DOI: 10.3390/informatics9030051.
X. Guo, K. Nerella, J. Dong, C. Qian, and Y. Chen. Understanding pitfalls and opportunities of applying heuristic evaluation methods to VR training systems An empirical study. International Journal of Human–Computer Interaction, 40(9), 2023. DOI: 10.1080/10447318.2022.2161238.
S. Stadler, H. Cornet, and F. Frenkler. Assessing heuristic evaluation in immersive virtual reality a case study on future guidance systems. Multimodal Technologies and Interaction, 7(2):19, 2023. DOI: 10.3390/mti7020019.
S. Gregor and A. R. Hevner. Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2):337–355, 2013. DOI: 10.25300/MISQ/2013/37.2.01.
A. R. Hevner, S. T. March, J. Park, and S. Ram. Design science in information systems research. MIS Quarterly, 28(1):75–105, 2004. DOI: 10.2307/25148625.
A. Bangor, P. T. Kortum, and J. T. Miller. An empirical evaluation of the System Usability Scale. International Journal of Human–Computer Interaction, 24(6):574–594, 2008. DOI: 10.1080/10447310802205776.
A. Wright. Education in the metaverse A new way of learning. Blog post, Spatial.io, Nov. 2021. Available [link]. Accessed 26 May 2025.
Spatial. Koreatech revolutionizes vocational training with interactive virtual learning. Case study, Spatial.io, 2025. Available [link]. Accessed 26 May 2025.
R. Brown, J. Recker, and S. West. Using virtual worlds for collaborative business process modeling. Business Process Management Journal, 17(3):546–564, 2011. DOI: 10.1108/14637151111136414.
C. Schwarz and Z. Wang. The role of digital twins in connected and automated vehicles. IEEE Intelligent Transportation Systems Magazine, 14(6):41–51, 2022. DOI: 10.1109/MITS.2021.3129524.
A. Kendrick. 10 usability heuristics applied to virtual reality. Nielsen Norman Group, Jul. 2021. Available [link]. Accessed 11 Jun. 2025.
Publicado
30/09/2025
Como Citar
FERREIRA, Yasmim Thasla Santos et al.
Assessing the Usability and Presence of an Immersive Virtual Environment: Opportunities and Challenges of Combining SUS, IPQ and Nielsen’s Heuristics. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 27. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 343-352.
