From Pixels to Interaction Graphs: Automated Usability Analytics in the SmartAPSUS Case Study
Resumo
Usability is a critical determinant of software quality, yet the post-test analytics phase remains a significant bottleneck for usability evaluation. Manually deriving metrics such as task completion time, state sequences, and click counts from session recordings is tedious, time-consuming, and dependent on expert evaluators. To mitigate this bottleneck, we propose UxMiner, a tool that automatically extracts quantitative usability metrics directly from screen recordings by combining Optical Character Recognition, computer vision, and a local Small Language Model. We demonstrate UxMiner’s feasibility through a running example on the SmartAPSUS public-health platform, where the tool processed an 8-minute session in approximately twice its duration, extracting 15 tasks, 181 interface states, 166 transitions, and 162 click events without any manual intervention.
Palavras-chave:
Usability Test, SmartAPSUS, UxMiner, Usability Metrics
Referências
William Albert and Tom Tullis. 2022. Measuring the user experience: Collecting, analyzing, and presenting UX metrics. Morgan Kaufmann.
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747 (2023). DOI: 10.48550/arXiv.2303.00747
Mateusz Barański, Jan Jasiński, Julitta Bartolewska, Stanisław Kacprzak, Marcin Witkowski, and Konrad Kowalczyk. 2025. Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio. In IEEE International Conference on Acoustics, Speech and Signal Processing. DOI: 10.1109/ICASSP49660.2025.10890105
Anders Bruun and Jan Stage. 2015. New Approaches to Usability Evaluation in Software Development: Barefoot and Crowdsourcing. Journal of Systems and Software 105 (2015), 40–53. DOI: 10.1016/j.jss.2015.03.043
John W. Castro, Ignacio Garnica, and Luis A. Rojas. 2022. Automated Tools for Usability Evaluation: A Systematic Mapping Study. In Social Computing and Social Media: Design, User Experience and Impact. DOI: 10.1007/978-3-031-05061-9_3
Joshua Daniel. 2024. Cursorlux. [link].
Richard O. Duda and Peter E. Hart. 1972. Use of the Hough transformation to detect lines and curves in pictures. Commun. ACM 15, 1 (1972). DOI: 10.1145/361237.361242
Juan M. Ferreira et al. 2020. Impact of usability mechanisms: An experiment on efficiency, effectiveness and user satisfaction. Information and Software Technology 117 (2020). DOI: 10.1016/j.infsof.2019.106195
Kasper Hornbæk. 2006. Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies 64, 2 (2006), 79–102. DOI: 10.1016/j.ijhcs.2005.06.002
Jessica L. Howe, Katharine T. Adams, A. Zachary Hettinger, and Raj M. Ratwani. 2018. Electronic Health Record Usability Issues and Potential Contribution to Patient Harm. JAMA 319, 12 (2018), 1276–1278. DOI: 10.1001/jama.2018.1171
International Organization for Standardization. 2018. Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. Standard ISO 9241-11:2018. International Organization for Standardization, Geneva, CH. [link]
Wolfgang Kluth, Karl-Heinz Krempels, and Christian Samsel. 2014. Automated Usability Testing for Mobile Applications. In International Conference on Web Information Systems and Technologies. DOI: 10.5220/0004985101490156
Allison Koenecke et al. 2024. Careless Whisper: Speech-to-Text Hallucination Harms. In ACM Conference on Fairness, Accountability, and Transparency. DOI: 10.1145/3630106.3658996
Jakob Nielsen and Kara Pernice. 2010. Eyetracking Web Usability. New Riders, Berkeley, CA.
Jim Nilsson and Tomas Akenine-Möller. 2020. Understanding ssim. arXiv preprint arXiv:2006.13846 (2020). DOI: 10.48550/arXiv.2006.13846
Suphakit Niwattanakul, Jatsada Singthongchai, Ekkachai Naenudorn, and Supachanun Wanapu. 2013. Using of Jaccard coefficient for keywords similarity. In international multiconference of engineers and computer scientists, Vol. 1. 380–384.
Ollama. 2026. Ollama: Run Llama 3, Mistral, Gemma, and other large language models locally. [link].
Online Appendix. 2026. nteraction Graphs for Automated Usability Analytics. DOI: 10.5281/zenodo.21264281.
Thiago Pinheiro, Mágno Gomes, Igor Vanderlei, Dimas Nascimento, Thiago Fabrício, Rodrigo Andrade, and Daliton Silva. 2025. SmartAPSUS: Integrando Dados, Previsão e Otimização para Fortalecer a Gestão da APS no Brasil. In Anais Estendidos do XL Simpósio Brasileiro de Bancos de Dados. DOI: 10.5753/sbbd_estendido.2025.247659
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In International Conference on Machine Learning.
Ronald Rivest. 1992. RFC1321: The MD5 message-digest algorithm. DOI: 10.17487/RFC1321
Maximilian Speicher, Brian Palmieri, Jialun Jiang, and Michael Nebeling. 2024. Enhancing UX Evaluation Through Collaboration with Conversational AI Assistants: Effects of Proactive Dialogue and Timing. InConference on Human Factors in Computing Systems. ACM, Honolulu, HI, USA, 1–17. DOI: 10.1145/3613904.3642168
An Yang et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115v2 (2025). DOI: 10.48550/arXiv.2412.15115
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747 (2023). DOI: 10.48550/arXiv.2303.00747
Mateusz Barański, Jan Jasiński, Julitta Bartolewska, Stanisław Kacprzak, Marcin Witkowski, and Konrad Kowalczyk. 2025. Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio. In IEEE International Conference on Acoustics, Speech and Signal Processing. DOI: 10.1109/ICASSP49660.2025.10890105
Anders Bruun and Jan Stage. 2015. New Approaches to Usability Evaluation in Software Development: Barefoot and Crowdsourcing. Journal of Systems and Software 105 (2015), 40–53. DOI: 10.1016/j.jss.2015.03.043
John W. Castro, Ignacio Garnica, and Luis A. Rojas. 2022. Automated Tools for Usability Evaluation: A Systematic Mapping Study. In Social Computing and Social Media: Design, User Experience and Impact. DOI: 10.1007/978-3-031-05061-9_3
Joshua Daniel. 2024. Cursorlux. [link].
Richard O. Duda and Peter E. Hart. 1972. Use of the Hough transformation to detect lines and curves in pictures. Commun. ACM 15, 1 (1972). DOI: 10.1145/361237.361242
Juan M. Ferreira et al. 2020. Impact of usability mechanisms: An experiment on efficiency, effectiveness and user satisfaction. Information and Software Technology 117 (2020). DOI: 10.1016/j.infsof.2019.106195
Kasper Hornbæk. 2006. Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies 64, 2 (2006), 79–102. DOI: 10.1016/j.ijhcs.2005.06.002
Jessica L. Howe, Katharine T. Adams, A. Zachary Hettinger, and Raj M. Ratwani. 2018. Electronic Health Record Usability Issues and Potential Contribution to Patient Harm. JAMA 319, 12 (2018), 1276–1278. DOI: 10.1001/jama.2018.1171
International Organization for Standardization. 2018. Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. Standard ISO 9241-11:2018. International Organization for Standardization, Geneva, CH. [link]
Wolfgang Kluth, Karl-Heinz Krempels, and Christian Samsel. 2014. Automated Usability Testing for Mobile Applications. In International Conference on Web Information Systems and Technologies. DOI: 10.5220/0004985101490156
Allison Koenecke et al. 2024. Careless Whisper: Speech-to-Text Hallucination Harms. In ACM Conference on Fairness, Accountability, and Transparency. DOI: 10.1145/3630106.3658996
Jakob Nielsen and Kara Pernice. 2010. Eyetracking Web Usability. New Riders, Berkeley, CA.
Jim Nilsson and Tomas Akenine-Möller. 2020. Understanding ssim. arXiv preprint arXiv:2006.13846 (2020). DOI: 10.48550/arXiv.2006.13846
Suphakit Niwattanakul, Jatsada Singthongchai, Ekkachai Naenudorn, and Supachanun Wanapu. 2013. Using of Jaccard coefficient for keywords similarity. In international multiconference of engineers and computer scientists, Vol. 1. 380–384.
Ollama. 2026. Ollama: Run Llama 3, Mistral, Gemma, and other large language models locally. [link].
Online Appendix. 2026. nteraction Graphs for Automated Usability Analytics. DOI: 10.5281/zenodo.21264281.
Thiago Pinheiro, Mágno Gomes, Igor Vanderlei, Dimas Nascimento, Thiago Fabrício, Rodrigo Andrade, and Daliton Silva. 2025. SmartAPSUS: Integrando Dados, Previsão e Otimização para Fortalecer a Gestão da APS no Brasil. In Anais Estendidos do XL Simpósio Brasileiro de Bancos de Dados. DOI: 10.5753/sbbd_estendido.2025.247659
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In International Conference on Machine Learning.
Ronald Rivest. 1992. RFC1321: The MD5 message-digest algorithm. DOI: 10.17487/RFC1321
Maximilian Speicher, Brian Palmieri, Jialun Jiang, and Michael Nebeling. 2024. Enhancing UX Evaluation Through Collaboration with Conversational AI Assistants: Effects of Proactive Dialogue and Timing. InConference on Human Factors in Computing Systems. ACM, Honolulu, HI, USA, 1–17. DOI: 10.1145/3613904.3642168
An Yang et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115v2 (2025). DOI: 10.48550/arXiv.2412.15115
Publicado
11/09/2026
Como Citar
VANDERLEI, Igor; ANDRADE, Rodrigo; GUIMARÃES, Allysson; EMÍDIO, Fernando; AQUINO, Gabriel; VANDERLEI, Miguel; NASCIMENTO, Dimas Cassimiro.
From Pixels to Interaction Graphs: Automated Usability Analytics in the SmartAPSUS Case Study. In: WORKSHOP BRASILEIRO DE ENGENHARIA DE SOFTWARE INTELIGENTE (ISE), 5. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 7-12.
DOI: https://doi.org/10.5753/ise.2026.32466.
