Using Networked Ontologies to support UX Evaluation in Immersive Context


Over the past few years, new interactive systems such as immersive technologies have gradually permeated our daily lives and found adoption across various fields. Immersive technologies provide users with immersive experiences. Assessing and modeling the quality of such experiences has become a trending topic in HCI, and UX is a key quality attribute in this context. When it comes to immersive experiences, evaluating UX is particularly challenging because the user should not be interrupted to provide feedback. In this paper, we propose using networked ontologies to support evaluating immersive experiences. We have explored using ontologies from an ontology network addressing the HCI domain to develop a tool that supports UX experts evaluating such experiences based on data recorded in interaction logs. We used the ontology-based tool to evaluate the UX of an immersive application that supports collaborative music composition. The tool extracted data from the application interaction logs applied UX metrics, and provided consolidated data and information in graphs and tables. We conducted a study and collected feedback from the tool developer and three UX experts who used the tool. Results showed that using networked ontologies to develop a tool to support UX evaluation is feasible and valuable. In summary, the ontologies helped at the conceptual level by offering a basis to define the system's structural model and at the implementation level by assigning semantics to data to make inferences about UX. Based on the UX experts' perceptions, the tool was considered a promising system, beneficial, helpful, and easy to use.

Palavras-chave: User Experience, UX Evaluation, Immersive Experience, Ontology, Ontology Network


William Albert and Thomas Tullis. 2013. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (2th ed.). Morgan Kaufmann, Boston, USA. 322 pages. DOI: 10.1016/C2011-0-00016-9

Mauro Amazonas, Thais Castro, Rosiane De Freitas, and Bruno Gadelha. 2019. Composing through Interaction: a framework for collaborative music composition based on human interaction on public spaces. In Proceedings of Brazilian Symposium on Musical Computing (SBCM 2019). Sociedade Brasileira de Computação - SBC, Porto Alegre, RS, Brasil, 42–49. DOI: 10.5753/sbcm.2019.10421

Jeferson Arango-López, Francisco Luis Gutiérrez Vela, Cesar Alberto Collazos, and Fernando Moreira. 2018. Modeling and Defining the Pervasive Games and Its Components from a Perspective of the Player Experience. In Proceedings of the Trends and Advances in Information Systems and Technologies. WorldCIST’18 2018. Advances in Intelligent Systems and Computing, vol 746, Álvaro Rocha, Hojjat Adeli, Luís Paulo Reis, and Sandra Costanzo (Eds.). Springer International Publishing, Cham, 625–635. DOI: 10.1007/978-3-319-77712-2_58

Maxim Bakaev, Sebastian Heil, Nikita Perminov, and Martin Gaedke. 2019. Integration Platform for Metric-Based Analysis of Web User Interfaces. In Proceedings of the International Conference on Web Engineering. ICWE 2019. Lecture Notes in Computer Science, vol 11496. Springer International Publishing, Cham, 525–529. DOI: 10.1007/978-3-030-19274-7_39

Simone Diniz Junqueira Barbosa, Bruno Santana da Silva, Milene Selbach Silveira, Isabela Gasparini, Ticianne Darin, and Gabriel Diniz Junqueira Barbosa. 2021. Interação Humano-Computador e Experiência do Usuário. Autopublicação.

Monalessa Barcellos, Ricardo de A. Falbo, and V G V Frauches. 2014. Towards a measurement ontology pattern language. Proceedings of the 1st Joint Workshop ONTO.COM / ODISE on Ontologies in Conceptual Modeling and Information Systems Engineering 1301 (2014), 14.

Monalessa Perini Barcellos, Ricardo de Almeida Falbo, and Ana Regina Rocha. 2010. Establishing a Well-Founded Conceptualization about Software Measurement in High Maturity Levels. In Proceedings of 2010 Seventh International Conference on the Quality of Information and Communications Technology. IEEE, Porto, Portugal, 467–472. DOI: 10.1109/QUATIC.2010.84

Coral Calero, Francisco Ruiz, and Mario Piattini. 2006. Ontologies for Software Engineering and Software Technology. Springer-Verlag, Berlin, Heidelberg. 354 pages.

Roberto Carraretto and Joao Paulo A. Almeida. 2012. Separating Ontological and Informational Concerns: Towards a Two-Level Model-Driven Approach. In Proceedings of 2012 IEEE 16th International Enterprise Distributed Object Computing Conference Workshops. IEEE, Beijing, China, 29–37. DOI: 10.1109/EDOCW.2012.14

John Millar Carroll. 2014. Human Computer Interaction (HCI). In The Encyclopedia of Human-Computer Interaction (2nd ed.), Mads Soegaard and Rikke Friis Dam (Eds.). The Interaction Design Foundation, Aarhus, Denmark, Chapter 2, 21–61.

Simone Dornelas Costa. 2022. An Ontology Network to support Knowledge Representation and Semantic Interoperability in the HCI Domain. Ph. D. Dissertation. Federal University of Espírito Santo, Vitória, Espírito Santo.

Simone Dornelas Costa, Monalessa Perini Barcellos, and Ricardo de Almeida Falbo. 2021. Ontologies in human–computer interaction: A systematic literature review. Applied Ontology 16, 4 (oct 2021), 1–32. DOI: 10.3233/AO-210255

Simone Dornelas Costa, Monalessa Perini Barcellos, Ricardo de Almeida Falbo, and Murillo Vasconcelos Henriques Bittencourt Castro. 2020. Towards an Ontology Network on Human-Computer Interaction. In Proceedings of the 39th International Conference on Conceptual Modeling, Gillian Dobbie, Ulrich Frank, Gerti Kappel, Stephen W Liddle, and Heinrich C Mayr (Eds.). Springer International Publishing, Cham, 331–341. DOI: 10.1007/978-3-030-62522-1_24

Simone Dornelas Costa, Monalessa Perini Barcellos, Ricardo de Almeida Falbo, Tayana Conte, and Káthia M. de Oliveira. 2022. A core ontology on the Human–Computer Interaction phenomenon. Data & Knowledge Engineering 138 (mar 2022), 101977. DOI: 10.1016/j.datak.2021.101977

Yasyn Elyusufi, Hamid Seghiouer, and Mohammed Amine Alimam. 2014. Building profiles based on ontology for recommendation custom interfaces. In 2014 International Conference on Multimedia Computing and Systems (ICMCS). IEEE, Marrakech, Morocco, 558–562. DOI: 10.1109/ICMCS.2014.6911166

Christina Feilmayr and Wolfram Wöß. 2016. An analysis of ontologies and their success factors for application to business. Data & Knowledge Engineering 101 (2016), 1–23. DOI: 10.1016/j.datak.2015.11.003

Nicola Guarino, Daniel Oberle, and Steffen Staab. 2009. What Is an Ontology? In Handbook on Ontologies. Springer Berlin Heidelberg, Berlin, Heidelberg, 1–17. DOI: 10.1007/978-3-540-92673-3_0

Giancarlo Guizzardi. 2005. Ontological foundations for structural conceptual models. Ph. D. Dissertation. University of Twente.

Giancarlo Guizzardi. 2007. On Ontology, Ontologies, Conceptualizations, Modeling Languages, and (Meta)Models. In Proceedings of the Frontiers in Artificial Intelligence and Applications, Databases and Information Systems IV, Olegas Vasilecas, Johan Edler, and Albertas Caplinskas (Eds.). IOS Press, Amsterdam, The Netherlands, The Netherlands, 18–39. [link]

Hans-Jörg Happel and Stefan Seedorf. 2006. Applications of Ontologies in Software Engineering. Proceedings of the 2nd International Workshop on Semantic Web Enabled Software Engineering (SWESE 2006), held at the 5th International Semantic Web Conference (ISWC 2006) (2006), 14.

Hala Magdy Hassan and Galal Hassan Galal-Edeen. 2017. From usability to user experience. In 2017 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS). IEEE, Okinawa, Japan, 216–222. DOI: 10.1109/ICIIBMS.2017.8279761

Marc Hassenzahl. 2008. User experience (UX) towards an experiential perspective on product quality. In Proceedings of the 20th Conference on l’Interaction Homme-Machine. Association for Computing Machinery, New York, NY, USA, 11–15. DOI: 10.1145/1512714.1512717

Marc Hassenzahl and Noam Tractinsky. 2006. User experience - a research agenda. Behaviour & Information Technology 25, 2 (2006), 91–97. DOI: 10.1080/01449290500330331

ISO. 2018. Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. Standard ISO 9241-11:2018. International Organization for Standardization. 36 pages. [link]

Effie Lai-Chong Law, Virpi Roto, Marc Hassenzahl, Arnold POS Vermeeren, and Joke Kort. 2009. Understanding, scoping and defining user experience: a survey approach. In Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 719–728. DOI: 10.1145/1518701.1518813

Leonardo Marques, Mauro Amazonas, Thais Castro, Willian Assuncao, Luciana Zaina, Bruno Gadelha, and Tayana Conte. 2020. UX-Trek: A post-interaction journey from immersive experience logs. In Proceedings of the XIX Brazilian Symposium on Human Factors in Computing Systems (IHC’20). ACM, Diamantina, Brazil, 6. DOI: 10.1145/3424953.3426547

Leonardo Marques, Patrícia Gomes Matsubara, Walter Takashi Nakamura, Bruna Moraes Ferreira, Igor Scaliante Wiese, Bruno Freitas Gadelha, Luciana Martinez Zaina, David Redmiles, and Tayana Uchôa Conte. 2021. Understanding UX better: A new technique to go beyond emotion assessment. Sensors 21, 21 (2021), 7183.

Nesrine Mezhoudi and Jean Vanderdonckt. 2015. A user’s feedback ontology for context-aware interaction. In Proceedings of the 2015 2nd World Symposium on Web Applications and Networking (WSWAN). IEEE, Sousse, Tunisia, 1–7. DOI: 10.1109/WSWAN.2015.7210331

Lukas Nagel, Oliver Karras, and Jil Klunder. 2021. Ontology-Based Software Graphs for Supporting Code Comprehension During Onboarding. In Proceedings of 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 158–165. DOI: 10.1109/SEAA53835.2021.00028

Walter T Nakamura, Edson Cesar de Oliveira, Elaine HT de Oliveira, David Redmiles, and Tayana Conte. 2022. What factors affect the UX in mobile apps? a systematic mapping study on the analysis of app store reviews. Journal of Systems and Software 193, C (2022), 111462. DOI: 10.1016/j.jss.2022.111462

Stefan Negru and Sabin Buraga. 2013. A Knowledge-Based Approach to the User-Centered Design Process. In Proceedings of the Knowledge Discovery, Knowledge Engineering and Knowledge Management. IC3K 2012. Communications in Computer and Information Science, vol 415., Ana Fred, Jan L G Dietz, Kecheng Liu, and Joaquim Filipe (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 165–178. DOI: 10.1007/978-3-642-54105-6_11

Stefan Negru and Sabin C Buraga. 2012. Persona Modeling Process - From Microdata-based Templates to Specific Web Ontologies. In Proceedings of the International Conference on Knowledge Engineering and Ontology Development, Joaquim Filipe and Jan Dietz (Eds.). SciTePress - Science and and Technology Publications, Barcelona, Spain, 34–42. DOI: 10.5220/0004127000340042

Jakob Nielsen. 1992. Finding usability problems through heuristic evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems. 373–380.

Heather L O’Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies 112 (2018), 28–39.

Helen Petrie and Nigel Bevan. 2009. The Evaluation of Accessibility, Usability, and User Experience. The universal access handbook 1 (jun 2009), 1–16. DOI: 10.1201/9781420064995-c20

Ingrid Pettersson, Florian Lachner, Anna-Katharina Frison, Andreas Riener, and Andreas Butz. 2018. A bermuda triangle? A review of method application and triangulation in user experience evaluation. In Proceedings of the 2018 CHI conference on human factors in computing systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–16. DOI: 10.1145/3173574.3174035

Jennifer Preece, Helen Sharp, and Yvonne Rogers. 2019. Interaction Design: Beyond Human-Computer Interaction (5th ed.). John Wiley & Sons, USA. 656 pages.

Luis Rivero and Tayana Conte. 2017. A systematic mapping study on research contributions on UX evaluation technologies. In Proceedings of the XVI Brazilian symposium on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1–10. DOI: 10.1145/3160504.3160512

Fabiano Borges Ruy, Ricardo de Almeida Falbo, Monalessa Perini Barcellos, Simone Dornelas Costa, and Giancarlo Guizzardi. 2016. SEON: A Software Engineering Ontology Network. In Proceedings of the Knowledge Engineering and Knowledge Management, Eva Blomqvist, Paolo Ciancarini, Francesco Poggi, and Fabio Vitali (Eds.). Springer International Publishing, Cham, 527–542. DOI: 10.1007/978-3-319-49004-5_34

Paulo Sérgio Santos Júnior, Monalessa Perini Barcellos, Ricardo de Almeida Falbo, and João Paulo A. Almeida. 2021. From a Scrum Reference Ontology to the Integration of Applications for Data-Driven Software Development. Information and Software Technology 136 (aug 2021), 106570. DOI: 10.1016/j.infsof.2021.106570

Steffen Staab and Rudi Studer. 2004. Handbook on Ontologies (1 ed.). Springer Berlin Heidelberg, Berlin, Heidelberg. XV, 660 pages. DOI: 10.1007/978-3-540-24750-0

Rudi Studer, V.Richard Benjamins, and Dieter Fensel. 1998. Knowledge engineering: Principles and methods. Data & Knowledge Engineering 25, 1-2 (mar 1998), 161–197. DOI: 10.1016/S0169-023X(97)00056-6

Mari Carmen Suárez-Figueroa, Asunción Gómez-Pérez, Enrico Motta, and Aldo Gangemi. 2012. Ontology Engineering in a Networked World. Springer Berlin Heidelberg, Berlin, Heidelberg. XII, 444 pages. DOI: 10.1007/978-3-642-24794-1

Ayoung Suh and Jane Prophet. 2018. The state of immersive technology research: A literature analysis. Computers in Human Behavior 86 (2018), 77–90. DOI: 10.1016/j.chb.2018.04.019

W3C. 2015. World Wide Web Consortium (W3C), Semantic Web. [link] Last accessed on 2023-02-03.
COSTA, Simone Dornelas; MANSO, Carolina De Freitas; MARQUES, Leonardo Carneiro; GADELHA, Bruno Freitas; CONTE, Tayana Uchôa; BARCELLOS, Monalessa Perini. Using Networked Ontologies to support UX Evaluation in Immersive Context. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2023 .