A Proposal for a Multimodal Framework for User Experience Evaluation in Interactive Web Systems
Abstract
Evaluating User Experience (UX) in different application domains is complex due to its multidisciplinary nature. Therefore, this study aims to identify requirements and propose a multimodal framework for evaluating UX in interactive Web systems to make it more holistic and comprehensive considering different domains. We applied a research method based on two phases: (I) the Exploratory Phase, and (II) the Framework Design Phase. We elicited 14 requirements and used them for an initial design of the multimodal framework proposal, which included guidelines, tools, and other research artifacts for evaluating UX in interactive Web systems in different domains.
References
Aviz, I. L., Souza, K. E., Ribeiro, E., de Mello Junior, H., e Seruffo, M. C. d. R. (2019). Comparative study of user experience evaluation techniques based on mouse and gaze tracking. In Proceedings of the 25th Brazillian symposium on multimedia and the Web, p. 53–56, New York, NY, USA. ACM.
Barbosa, S. D. J., Silva, B. S. d., Silveira, M. S., Gasparini, I., Darin, T., e Barbosa, G. D. J. (2021). Interação Humano-Computador e Experiência do Usuário. Autopublicação.
Cambridge (2024). Multimodal. in: Cambridge dictionary. [link]. Acessado: 26/03/2024.
Corbin, J. e Strauss, A. (2014). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, New York, NY, USA, 4 edition.
DIS, I. (2019). 9241-210: 2019. ergonomics of human system interaction-part 210: Human-centred design for interactive systems. International Standardization Organization (ISO). Switzerland. Disponível em: [link].
Kremer, S. e Lindemann, U. (2015). A framework for understanding, communicating and evaluating user experience potentials. In 20th International Conference on Engineering Design (ICED15), p. 1–11, Milan, Italy. Politecnico di Milano, Design Society.
Larigaldie, N., Dreneva, A., e Orquin, J. L. (2024). eyescrollr: A software method for reproducible mapping of eye-tracking data from scrollable web pages. Behavior Research Methods, p. 1–16.
Lima, D. T., Moura, F. R. T., Alves, A. V. N., Parracho, T. d. M., Zacarias, R. O., Santos, R. P. d., e Seruffo, M. C. d. R. (2022a). Ux-tracking: Web and multimodal tool for user experience evaluation. In Anais Estendidos do XXVIII Simpósio Brasileiro de Sistemas Multimídia e Web, p. 107–110, Porto Alegre, RS, Brasil. SBC.
Lima, D. T., Moura, F. R. T., Souza, K. E. S. d., Paulino, R. d. C. R., e Seruffo, M. C. d. R. (2022b). Rastreamento de mouse para avaliação de experiência do usuário em portais de notícias: Um estudo de caso. In Freitas, L. C. d., editor, Collection: Applied Computer Engineering 2, p. 74–86, Ponta Grossa, PR, Brasil. Atena Editora.
Martinelli, S., Lopes, L., e Zaina, L. (2024). Ux research practices related to long-term ux: A systematic literature review. Information and Software Technology, 170:107431.
Norman, D. e Nielsen, J. (2006). The definition of user experience (ux). [link].
Ntoa, S., Margetis, G., Antona, M., e Stephanidis, C. (2021). User experience evaluation in intelligent environments: A comprehensive framework. Technologies, 9(2).
Olson, G. M. e Olson, J. S. (2003). Human-computer interaction: Psychological aspects of the human use of computing. Annual review of psychology, 54(1):491–516.
Paulino, R. d. C. R., Seruffo, M. C. d. R., Empinotti, M. L., de Souza, K. E. S., e Pimenta, A. C. (2021). Análise da experiência do usuário (ux) de narrativa transmídia através de mouse-tracking. Comunicação e Inovação, 22(50):41–61.
Rodrigues, M. E. M., Moura, K. H. S., Castelo Branco, K., Lelli, V., Viana, W., Andrade, R. M. C., e Santos, I. S. (2022). Medication time? a user experience evaluation of mobile applications targeting people with diabetes. In Proceedings of the Brazilian Symposium on Multimedia and the Web, WebMedia ’22, p. 258–266, New York, NY, USA. ACM.
Rogers, Y., Sharp, H., e Preece, J. (2011). Interaction design: beyond human-computer interaction. John Wiley & Sons, Hoboken, NJ, USA, 3 edition.
Santos, R. P. (2016). Managing and Monitoring Software Ecosystem to Support Demand and Solution Analysis. Tese de doutorado, COPPE/UFRJ, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brasil.
Santos, S. S. d. e Rodrigues, K. R. d. H. (2022). A systematic mapping study ofnbsp;emotional response evaluation instruments. In HCI International 2022 Late Breaking Papers. Design, User Experience and Interaction: 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, p. 302–317, Berlin, Heidelberg. Springer-Verlag.
Sari, J. N., Nugroho, L. E., Ferdiana, R., e Santosa, P. I. (2018). Evaluation of fixation duration accuracy in determining selected product on e-commerce. In 2018 10th International Conference on Information Technology and Electrical Engineering (ICITEE), p. 146–151. IEEE.
Sommerville, I. (2011). Engenharia de software. Pearson Prentice Hall, Hoboken, NJ, USA, 9th edition.
Sousa, A. e Valentim, N. (2020). Projetando a usabilidade e a experiência do usuário com a técnica uxug-ap: um estudo exploratório. In Anais do V Workshop sobre Aspectos Sociais, Humanos e Econômicos de Software, p. 21–30, Porto Alegre, RS, Brasil. SBC.
Souza, K. E. S. d., Aviz, I. L. d., Mello, H. D. d., Figueiredo, K., Vellasco, M. M. B. R., Costa, F. A. R., e Seruffo, M. C. d. R. (2022). An evaluation framework for user experience using eye tracking, mouse tracking, keyboard input, and artificial intelligence: A case study. v. 38, p. 41–61, Abingdon, Oxon, UK. Taylor & Francis.
Souza, K. E. S. d., Zacarias, R. O., Seruffo, M. C. d. R., e Santos, R. P. d. (2020). T2-uxt: A tool to support transparency evaluation in software ecosystems portals. In Proceedings of the 34th Brazilian Symposium on Software Engineering, p. 415–420, New York, NY, USA. ACM.
Staden, C. J. v., Biljon, J. A., e Kroeze, J. H. (2017). Using a user experience evaluation framework for emoderation. In 2017 Conference on Information Communication Technology and Society (ICTAS), p. 1–6. IEEE.
Sun, F.-S., Kong, X., e Weng, Y.-H. (2018). An interactive web system for group project management and peer evaluation. In Proceedings of the 2nd International Conference on Information System and Data Mining, ICISDM ’18, p. 103–106, New York, NY, USA. ACM.
Wiegers, K. E. e Beatty, J. (2013). Software Requirements. Microsoft Press, USA, 3 edition.
Zacarias, R. O., Lima, D. T., Seruffo, M. C. d. R., e Santos, R. P. d. (2022). Perspectivas do monitoramento de experiência de usuário em diferentes domínios com ferramenta de captura multimodal. In Anais do XIII Workshop sobre Aspectos da Interação Humano-Computador para a Web Social, p. 48–55, Porto Alegre, RS, Brasil. SBC.
Barbosa, S. D. J., Silva, B. S. d., Silveira, M. S., Gasparini, I., Darin, T., e Barbosa, G. D. J. (2021). Interação Humano-Computador e Experiência do Usuário. Autopublicação.
Cambridge (2024). Multimodal. in: Cambridge dictionary. [link]. Acessado: 26/03/2024.
Corbin, J. e Strauss, A. (2014). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, New York, NY, USA, 4 edition.
DIS, I. (2019). 9241-210: 2019. ergonomics of human system interaction-part 210: Human-centred design for interactive systems. International Standardization Organization (ISO). Switzerland. Disponível em: [link].
Kremer, S. e Lindemann, U. (2015). A framework for understanding, communicating and evaluating user experience potentials. In 20th International Conference on Engineering Design (ICED15), p. 1–11, Milan, Italy. Politecnico di Milano, Design Society.
Larigaldie, N., Dreneva, A., e Orquin, J. L. (2024). eyescrollr: A software method for reproducible mapping of eye-tracking data from scrollable web pages. Behavior Research Methods, p. 1–16.
Lima, D. T., Moura, F. R. T., Alves, A. V. N., Parracho, T. d. M., Zacarias, R. O., Santos, R. P. d., e Seruffo, M. C. d. R. (2022a). Ux-tracking: Web and multimodal tool for user experience evaluation. In Anais Estendidos do XXVIII Simpósio Brasileiro de Sistemas Multimídia e Web, p. 107–110, Porto Alegre, RS, Brasil. SBC.
Lima, D. T., Moura, F. R. T., Souza, K. E. S. d., Paulino, R. d. C. R., e Seruffo, M. C. d. R. (2022b). Rastreamento de mouse para avaliação de experiência do usuário em portais de notícias: Um estudo de caso. In Freitas, L. C. d., editor, Collection: Applied Computer Engineering 2, p. 74–86, Ponta Grossa, PR, Brasil. Atena Editora.
Martinelli, S., Lopes, L., e Zaina, L. (2024). Ux research practices related to long-term ux: A systematic literature review. Information and Software Technology, 170:107431.
Norman, D. e Nielsen, J. (2006). The definition of user experience (ux). [link].
Ntoa, S., Margetis, G., Antona, M., e Stephanidis, C. (2021). User experience evaluation in intelligent environments: A comprehensive framework. Technologies, 9(2).
Olson, G. M. e Olson, J. S. (2003). Human-computer interaction: Psychological aspects of the human use of computing. Annual review of psychology, 54(1):491–516.
Paulino, R. d. C. R., Seruffo, M. C. d. R., Empinotti, M. L., de Souza, K. E. S., e Pimenta, A. C. (2021). Análise da experiência do usuário (ux) de narrativa transmídia através de mouse-tracking. Comunicação e Inovação, 22(50):41–61.
Rodrigues, M. E. M., Moura, K. H. S., Castelo Branco, K., Lelli, V., Viana, W., Andrade, R. M. C., e Santos, I. S. (2022). Medication time? a user experience evaluation of mobile applications targeting people with diabetes. In Proceedings of the Brazilian Symposium on Multimedia and the Web, WebMedia ’22, p. 258–266, New York, NY, USA. ACM.
Rogers, Y., Sharp, H., e Preece, J. (2011). Interaction design: beyond human-computer interaction. John Wiley & Sons, Hoboken, NJ, USA, 3 edition.
Santos, R. P. (2016). Managing and Monitoring Software Ecosystem to Support Demand and Solution Analysis. Tese de doutorado, COPPE/UFRJ, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brasil.
Santos, S. S. d. e Rodrigues, K. R. d. H. (2022). A systematic mapping study ofnbsp;emotional response evaluation instruments. In HCI International 2022 Late Breaking Papers. Design, User Experience and Interaction: 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26 – July 1, 2022, Proceedings, p. 302–317, Berlin, Heidelberg. Springer-Verlag.
Sari, J. N., Nugroho, L. E., Ferdiana, R., e Santosa, P. I. (2018). Evaluation of fixation duration accuracy in determining selected product on e-commerce. In 2018 10th International Conference on Information Technology and Electrical Engineering (ICITEE), p. 146–151. IEEE.
Sommerville, I. (2011). Engenharia de software. Pearson Prentice Hall, Hoboken, NJ, USA, 9th edition.
Sousa, A. e Valentim, N. (2020). Projetando a usabilidade e a experiência do usuário com a técnica uxug-ap: um estudo exploratório. In Anais do V Workshop sobre Aspectos Sociais, Humanos e Econômicos de Software, p. 21–30, Porto Alegre, RS, Brasil. SBC.
Souza, K. E. S. d., Aviz, I. L. d., Mello, H. D. d., Figueiredo, K., Vellasco, M. M. B. R., Costa, F. A. R., e Seruffo, M. C. d. R. (2022). An evaluation framework for user experience using eye tracking, mouse tracking, keyboard input, and artificial intelligence: A case study. v. 38, p. 41–61, Abingdon, Oxon, UK. Taylor & Francis.
Souza, K. E. S. d., Zacarias, R. O., Seruffo, M. C. d. R., e Santos, R. P. d. (2020). T2-uxt: A tool to support transparency evaluation in software ecosystems portals. In Proceedings of the 34th Brazilian Symposium on Software Engineering, p. 415–420, New York, NY, USA. ACM.
Staden, C. J. v., Biljon, J. A., e Kroeze, J. H. (2017). Using a user experience evaluation framework for emoderation. In 2017 Conference on Information Communication Technology and Society (ICTAS), p. 1–6. IEEE.
Sun, F.-S., Kong, X., e Weng, Y.-H. (2018). An interactive web system for group project management and peer evaluation. In Proceedings of the 2nd International Conference on Information System and Data Mining, ICISDM ’18, p. 103–106, New York, NY, USA. ACM.
Wiegers, K. E. e Beatty, J. (2013). Software Requirements. Microsoft Press, USA, 3 edition.
Zacarias, R. O., Lima, D. T., Seruffo, M. C. d. R., e Santos, R. P. d. (2022). Perspectivas do monitoramento de experiência de usuário em diferentes domínios com ferramenta de captura multimodal. In Anais do XIII Workshop sobre Aspectos da Interação Humano-Computador para a Web Social, p. 48–55, Porto Alegre, RS, Brasil. SBC.
Published
2024-07-21
How to Cite
ZACARIAS, Rodrigo; PARRACHO, Thiago; MOURA, Flávio; LIMA, Danilo; SANTOS, Suzane; RODRIGUES, Kamila; SERUFFO, Marcos; SANTOS, Rodrigo.
A Proposal for a Multimodal Framework for User Experience Evaluation in Interactive Web Systems. In: PROCEEDINGS OF WORKSHOP ON SOCIAL, HUMAN AND ECONOMIC ASPECTS OF SOFTWARE (WASHES), 9. , 2024, Brasília/DF.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 94-105.
ISSN 2763-874X.
DOI: https://doi.org/10.5753/washes.2024.2842.
