LLM-Based Usability Inspection: A Systematic Mapping of Automation Strategies, Evidence, and Validation Practices
Resumo
Usability inspection is an important activity for identifying interaction problems, and recent advances in Large Language Models (LLMs) have created new opportunities to support or automate this process. However, it remains unclear how LLMs have been applied to usability inspection, which evidence and inspection criteria they rely on, and how their outputs are validated. This paper presents a systematic literature mapping of LLM-based usability and UX inspection. From 401 publications retrieved from Scopus, ACM Digital Library, IEEE Xplore, and SBC OpenLib, 14 primary studies published between 2023 and 2025 were selected. The studies were analyzed with respect to LLMs, prompting strategies, input artifacts, inspection criteria, validation approaches, and reported limitations. The results show a predominance of GPT-4-family models and fully automated workflows, with screenshots and structured UI representations as the most common input artifacts. Prompt engineering, particularly role prompting and chain-of-thought, is widely adopted, while validation practices remain heterogeneous. Recurring limitations include hallucinations, false positives, limited contextual judgment, and prompt dependence. These findings indicate that LLM-based usability inspection is an emerging topic in Intelligent Software Engineering, in which human-AI collaboration currently appears more realistic than full automation.
Palavras-chave:
Usability Inspection, LLM, UX Evaluation, Human-AI Collaboration, Software Quality, Intelligent Software Engineering
Referências
Alba Bisante, Venkata Srikanth Varma Datla, Emanuele Panizzi, Gabriella Trasciatti, and Stefano Zeppieri. 2024. Enhancing Interface Design with AI: An Exploratory Study on a ChatGPT-4-Based Tool for Cognitive Walkthrough Inspired Evaluations. In Proceedings of the 2024 International Conference on Advanced Visual Interfaces (Arenzano, Genoa, Italy) (AVI ’24). Association for Computing Machinery, New York, NY, USA, Article 41, 5 pages. DOI: 10.1145/3656650.3656676
Peitong Duan, Chin-Yi Cheng, Gang Li, Bjoern Hartmann, and Yang Li. 2024. UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST ’24). Association for Computing Machinery, New York, NY, USA, Article 46, 17 pages. DOI: 10.1145/3654777.3676381
Peitong Duan, Jeremy Warner, and Bjoern Hartmann. 2023. Towards Generating UI Design Feedback with LLMs. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 70, 3 pages. DOI: 10.1145/3586182.3615810
Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 6, 20 pages. DOI: 10.1145/3613904.3642782
Guilherme Guerino, Luiz Rodrigues, Bruna Capeleti, Rafael Ferreira Mello, André Freire, and Luciana Zaina. 2025. Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation. arXiv:2506.16345 [cs.HC] [link]
Nien-Lin Hsueh, Hsuen-Jen Lin, and Lien-Chi Lai. 2024. Applying Large Language Model to User Experience Testing. Electronics 13, 23 (2024). DOI: 10.3390/electronics13234633
International Organization for Standardization. 2018. ISO 9241-11:2018 — Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. Disponível em: [link].
Melody Y. Ivory and Marti A. Hearst. 2001. The State of the Art in Automating Usability Evaluation of User Interfaces. Comput. Surveys 33, 4 (2001), 470–516.
Eduard Kuric, Peter Demcak, Matus Krajcovic, and Jan Lang. 2025. Systematic Literature Review of Automation and Artificial Intelligence in Usability Issue Detection. arXiv preprint arXiv:2504.01415 (2025).
Jia Liu. 2025. AI-Powered Automated and Remote UX Evaluation Methods: A Systematic Literature Review. In Proceedings of the 43rd ACM International Conference on Design of Communication (SIGDOC ’25). Association for Computing Machinery, New York, NY, USA, 10–16. DOI: 10.1145/3711670.3764614
Yuwen Lu, Yuewen Yang, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. 2024. AI Assistance for UX: A Literature Review Through Human-Centered AI. arXiv preprint arXiv:2402.06089 (2024).
Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Zheshen Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. 2025. UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25). ACM, 1–12. DOI: 10.1145/3706599.3719729
Jakob Nielsen and Robert L. Mack. 1994. Usability Inspection Methods. John Wiley & Sons, New York, NY, USA.
Donald A. Norman. 2006. O design do dia-a-dia. Rocco, Rio de Janeiro.
Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008. Systematic Mapping Studies in Software Engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (EASE). 68–77.
Ali Ebrahimi Pourasad and Walid Maalej. 2025. Does GenAI Make Usability Testing Obsolete? arXiv:2411.00634 [cs.SE] [link]
Simone Santos, Robson Alves, Francisco Oliveira, and Francisco Araujo. 2025. Investigating LLM-based tools to support Usability, Accessibility, User Experience in HCI activities: A Systematic Literature Mapping. In Proceedings of the 31st Brazilian Symposium on Multimedia and the Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 631–645. DOI: 10.5753/webmedia.2025.16160
Subin Shin, Jeesun Oh, and Sangwon Lee. 2025. Can LLMs See What I See? A Study on Five Prompt Engineering Techniques for Evaluating UX on a Shopping Site. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25). Association for ComputingMachinery, New York, NY, USA, Article 125, 7 pages. DOI: 10.1145/3706599.3720079
Jason Wu, Yi-Hao Peng, Amanda Li, Amanda Swearngin, Jeffrey P. Bigham, and Jeffrey Nichols. 2024. UIClip: A Data-driven Model for Assessing User Interface Design. arXiv:2404.12500 [cs.HC] [link]
Mingyuan Zhong, Ruolin Chen, Xia Chen, James Fogarty, and Jacob O. Wobbrock. 2025. ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 1163, 19 pages. DOI: 10.1145/3706598.3713797
Peitong Duan, Chin-Yi Cheng, Gang Li, Bjoern Hartmann, and Yang Li. 2024. UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST ’24). Association for Computing Machinery, New York, NY, USA, Article 46, 17 pages. DOI: 10.1145/3654777.3676381
Peitong Duan, Jeremy Warner, and Bjoern Hartmann. 2023. Towards Generating UI Design Feedback with LLMs. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 70, 3 pages. DOI: 10.1145/3586182.3615810
Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 6, 20 pages. DOI: 10.1145/3613904.3642782
Guilherme Guerino, Luiz Rodrigues, Bruna Capeleti, Rafael Ferreira Mello, André Freire, and Luciana Zaina. 2025. Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation. arXiv:2506.16345 [cs.HC] [link]
Nien-Lin Hsueh, Hsuen-Jen Lin, and Lien-Chi Lai. 2024. Applying Large Language Model to User Experience Testing. Electronics 13, 23 (2024). DOI: 10.3390/electronics13234633
International Organization for Standardization. 2018. ISO 9241-11:2018 — Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. Disponível em: [link].
Melody Y. Ivory and Marti A. Hearst. 2001. The State of the Art in Automating Usability Evaluation of User Interfaces. Comput. Surveys 33, 4 (2001), 470–516.
Eduard Kuric, Peter Demcak, Matus Krajcovic, and Jan Lang. 2025. Systematic Literature Review of Automation and Artificial Intelligence in Usability Issue Detection. arXiv preprint arXiv:2504.01415 (2025).
Jia Liu. 2025. AI-Powered Automated and Remote UX Evaluation Methods: A Systematic Literature Review. In Proceedings of the 43rd ACM International Conference on Design of Communication (SIGDOC ’25). Association for Computing Machinery, New York, NY, USA, 10–16. DOI: 10.1145/3711670.3764614
Yuwen Lu, Yuewen Yang, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. 2024. AI Assistance for UX: A Literature Review Through Human-Centered AI. arXiv preprint arXiv:2402.06089 (2024).
Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Zheshen Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. 2025. UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25). ACM, 1–12. DOI: 10.1145/3706599.3719729
Jakob Nielsen and Robert L. Mack. 1994. Usability Inspection Methods. John Wiley & Sons, New York, NY, USA.
Donald A. Norman. 2006. O design do dia-a-dia. Rocco, Rio de Janeiro.
Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008. Systematic Mapping Studies in Software Engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (EASE). 68–77.
Ali Ebrahimi Pourasad and Walid Maalej. 2025. Does GenAI Make Usability Testing Obsolete? arXiv:2411.00634 [cs.SE] [link]
Simone Santos, Robson Alves, Francisco Oliveira, and Francisco Araujo. 2025. Investigating LLM-based tools to support Usability, Accessibility, User Experience in HCI activities: A Systematic Literature Mapping. In Proceedings of the 31st Brazilian Symposium on Multimedia and the Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 631–645. DOI: 10.5753/webmedia.2025.16160
Subin Shin, Jeesun Oh, and Sangwon Lee. 2025. Can LLMs See What I See? A Study on Five Prompt Engineering Techniques for Evaluating UX on a Shopping Site. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25). Association for ComputingMachinery, New York, NY, USA, Article 125, 7 pages. DOI: 10.1145/3706599.3720079
Jason Wu, Yi-Hao Peng, Amanda Li, Amanda Swearngin, Jeffrey P. Bigham, and Jeffrey Nichols. 2024. UIClip: A Data-driven Model for Assessing User Interface Design. arXiv:2404.12500 [cs.HC] [link]
Mingyuan Zhong, Ruolin Chen, Xia Chen, James Fogarty, and Jacob O. Wobbrock. 2025. ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 1163, 19 pages. DOI: 10.1145/3706598.3713797
Publicado
11/09/2026
Como Citar
FARIAS, Ana Beatriz; ALBUQUERQUE, Danyllo; DANTAS FILHO, Emanuel; NETO, Ademar.
LLM-Based Usability Inspection: A Systematic Mapping of Automation Strategies, Evidence, and Validation Practices. In: WORKSHOP BRASILEIRO DE ENGENHARIA DE SOFTWARE INTELIGENTE (ISE), 5. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 25-30.
DOI: https://doi.org/10.5753/ise.2026.32618.
