User Experience Evaluation using Machine Learning and Facial Expressions: A Systematic Review

• Bianca Lima Santos USP
• Luciano A. Digiampietri USP

DOI: https://doi.org/10.5753/eniac.2024.245150

Resumo

This systematic review investigates the application of machine learning in facial expressions and emotion recognition within the realm of user experience (UX). The main objective is to identify advances in the state of the art regarding using facial expressions to detect emotions and, consequently, predict or improve user experience. The methodology provided a comprehensive analysis of existing literature, highlighting diverse definitions of UX and their implications for assessing user interactions with systems. Despite efforts to evaluate and enhance UX through various methodologies, few studies focus on predicting UX by integrating emotional states before interaction and user-reported experiences. This gap stems from the absence of a unified UX definition, complicating methodological standardization and result comparability across studies. Many reviewed works emphasize developing recommendation algorithms tailored to music, news, and other content to optimize UX through emotional data. The review identified the challenge of establishing a consistent framework for UX definition across research, revealing varied approaches using different datasets aimed at enhancing recommendations, improving user satisfaction, comparing perceived attitudes, and integrating with established questionnaires.

Palavras-chave: Machine Learning, User Experience, Emotion Recognition

Referências

Afriansyah, Y., Nugrahaeni, R. A., and Prasasti, A. L. (2021). Facial expression classification for user experience testing using k-nearest neighbor. In 2021 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT), pages 63–68.

Ashani Malsha, N. P. G., Dulmini Heshani, K., Ransara, R. K., Ayesh Bandara, D. M. D. D., Suriyaa Kumari, P. K., and Anuththara Kuruppu, T. (2021). Automated sinhala news platform based on machine learning and deep learning. In 2021 3rd International Conference on Advancements in Computing (ICAC), pages 134–139.

Bernhaupt, R. (2015). User Experience Evaluation Methods in the Games Development Life Cycle, pages 1–8. Springer International Publishing, Cham.

Chimienti, M., Danzi, I., Gattulli, V., Impedovo, D., Pirlo, G., and Veneto, D. (2022). Behavioral analysis for user satisfaction. In 2022 IEEE Eighth International Conference on Multimedia Big Data (BigMM), pages 113–119.

Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., and Taylor, J. (2001). Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 18(1):32–80.

Dias, B., Ivamoto, V., and Lima, C. (2023). Detecting faces in specific scenarios: Systematic literature review. In Anais do XX Encontro Nacional de Inteligência Artificial e Computacional, pages 540–554, Porto Alegre, RS, Brasil. SBC.

Eliyajer, G., Natarajan, B., Bhuvaneswari, R., and Elakkiya, R. (2023). A novel approach for song recommendation system using deep neural networks. In 2023 IEEE World Conference on Applied Intelligence and Computing (AIC), pages 382–387.

Gupta, M., Venkatesh, S. N., Suraskar, R., Praveena, K., Jegadesan, S., and Suneetha, S. (2023). Enhancing music recommendations with emotional insight: A facial expression approach in ai. In 2023 7th International Conference on Electronics, Communication and Aerospace Technology (ICECA), pages 450–457.

Isman, F. A., Prasasti, A. L., and Nugrahaeni, R. A. (2021). Expression classification for user experience testing using convolutional neural network. In 2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS), pages 1–6.

ISO 9241-210 (2010). Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems. Standard ISO 9241-210:2010, International Organization for Standardization.

Karimah, S. N., Phan, H., Miftakhurrokhmat, and Hasegawa, S. (2024). Design principle of an automatic engagement estimation system in a synchronous distance learning practice. IEEE Access, 12:25598–25611.

Koonsanit, K. and Nishiuchi, N. (2020). Classification of user satisfaction using facial expression recognition and machine learning. In 2020 IEEE REGION 10 CONFERENCE (TENCON), pages 561–566.

Kwon, S., Ahn, J., Choi, H., Jeon, J., Kim, D., Kim, H., and Kang, S. (2022). Analytical framework for facial expression on game experience test. IEEE Access, 10:104486–104497.

Liu, X. and Lee, K. (2018). Optimized facial emotion recognition technique for assessing user experience. In 2018 IEEE Games, Entertainment, Media Conference (GEM), pages 1–9.

Mabel Rani, A. J., S, M., Jothi Swaroopan, N. M., and Hari Kumar, K. (2023). Face emotion based music recommendation system using modified convolution neural network. In 2023 International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE), pages 1–6.

Qian, Y., Lu, J., Miao, Y., Ji, W., Jin, R., and Song, E. (2018). Aiem: Ai-enabled affective experience management. Future Generation Computer Systems, 89:438–445.

Selvi, A. S., S, A., Hariharan, M., and P, A. (2023). Emotune: Deep emotion detection and music recommendation system using mobilenetv3. In 2023 International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE), pages 1–6.

van Erven, R. C. G. a. S. and Canedo, E. D. (2023). Measurement of user’s satisfaction of digital products through emotion recognition. In Proceedings of the XXII Brazilian Symposium on Software Quality, SBQS ’23, page 62–71, New York, NY, USA. Association for Computing Machinery.

Veriscimo, E. d. S., Bernardes-Júnior, J. L., and Digiampietri, L. A. (2020). Evaluating User Experience in 3D Interaction: a Systematic Review. In Proceedings of the XVI Brazilian Symposium on Information Systems - SBSI, SBSI ’20, New York, NY, USA. Association for Computing Machinery.

Veriscimo, E. d. S., Bernardes-Júnior, J. L., and Digiampietri, L. A. (2021). Facial emotion recognition in ux evaluation: A systematic review. In Kurosu, M., editor, Human-Computer Interaction. Theory, Methods and Tools, pages 521–534, Cham. Springer International Publishing.