An Exploratory Study on Prompting Strategies for LLM-Assisted Heuristic Evaluation of Web User Interfaces

• Ana Beatriz Farias IFPB
• Danyllo Albuquerque IFPB
• Emanuel Dantas Filho IFPE
• Rohit Gheyi UFCG
• Francisco Petrônio Medeiros IFPB

DOI: https://doi.org/10.5753/sast.2026.30621

Resumo

Usability is a key non-functional requirement in interactive software systems, and heuristic evaluation can support the verification and validation of user interfaces by identifying violations of usability principles. However, this process still depends on human expertise, is subject to evaluator variability, and may be difficult to scale. Although recent advances in Large Language Models (LLMs) create opportunities for AI-assisted heuristic evaluation, there is limited empirical evidence on the reliability of LLM-generated diagnoses. This paper presents an empirical study of GPT-5.4-assisted heuristic evaluation using Nielsen’s heuristics (H), with diagnoses evaluated against an expert-derived ground truth.We evaluate zero-shot, fewshot, and chain-of-thought prompting on 12 web user interfaces, comparing model diagnoses with the reference evaluation using precision, recall, F1-score, and Cohen’s kappa. Few-shot prompting achieved the best precision-recall balance (F1=0.51, 𝜅=0.193), while chain-of-thought achieved the highest recall but produced more false positives. Heuristic-level results revealed distinct error patterns, with high recovery for H8 and H10 and frequent unsupported predictions for H4, H5, and H6, although these patterns are affected by uneven ground-truth prevalence. These findings suggest that prompting strategy is an important design consideration in LLM-assisted heuristic evaluation, while human validation remains necessary.

Palavras-chave: AI-Assisted Software Testing, Large Language Models, Prompt Engineering, Usability Inspection, Heuristic Evaluation, Non-Functional Requirements, User Interfaces

Referências

Issa Atoum, M. Baklizi, I. Alsmadi, A. Otoom, Taha Alhersh, Jafar Ababneh, Jameel Almalki, and S. M. Alshahrani. 2021. Challenges of Software Requirements Quality Assurance and Validation: A Systematic Literature Review. IEEE Access 9 (2021), 137613–137634. DOI: 10.1109/access.2021.3117989

Nathalia Barbosa, Paulo Borba, and Léuson M. P. da Silva. 2025. Detecting Semantic Conflicts with LLM-Generated Tests. In Proceedings of the 10th Brazilian Symposium on Systematic and Automated Software Testing (SAST 2025). Sociedade Brasileira de Computação, Recife, Brazil, 18–27.

Tom B Brown et al. 2020. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (2020), 1877–1901.

Larissa Chazette and Kurt Schneider. 2020. Explainability as a non-functional requirement: challenges and recommendations. Requirements Engineering 25, 4 (2020), 493–514.

Peitong Duan, Chin-yi Chen, Gang Li, Bjoern Hartmann, and Yang Li. 2024. UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. ACM, 1–17. DOI: 10.1145/3654777.3676381

Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. ACM, 1–20. DOI: 10.1145/3613904.3642782

Andrea Esposito, Giuseppe Desolda, and R. Lanzilotti. 2024. The fine line between automation and augmentation in website usability evaluation. Scientific Reports 14, 1 (2024), 10129. DOI: 10.1038/s41598-024-59616-0

Nayse Fagundes and Leopoldo Teixeira. 2025. Evaluating LLMs for Multimodal GUI Test Generation in Android Applications. In Proceedings of the 10th Brazilian Symposium on Systematic and Automated Software Testing (SAST 2025). Sociedade Brasileira de Computação, Recife, Brazil, 28–36.

Guilherme Guerino, Luiz Rodrigues, Bruna Capeleti, Rafael Ferreira Mello, André Freire, and Luciana Zaina. 2026. Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation. In Human-Computer Interaction – INTERACT 2025. Springer Nature Switzerland, Cham, 381–402.

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79. DOI: 10.1145/3695988

Nien-Lin Hsueh, Hsuen-Jen Lin, and Lien-Chi Lai. 2024. Applying Large Language Model to User Experience Testing. Electronics 13, 23 (2024). DOI: 10.3390/electronics13234633

International Organization for Standardization. 2018. ISO 9241-11:2018 — Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. Available at: [link]. Accessed: August 11, 2026.

Malik Asif Jilani, Iftikhar Azim Niaz, Saad Naeem Zafar, Faiz Ali Shah, Shujaat Ali, Dilawar Shah, and Muhammad Tahir. 2026. Usability Quality Model: An Enhancement of Dromey’s Model. Engineering Reports 8, 4 (2026), e70747.

J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159–174.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv:2107.13586 [cs.CL] [link]

Rian Melo, Pedro Simoes, Rohit Gheyi, Marcelo d’Amorim, Marcio Ribeiro, Gustavo Soares, Eduardo Almeida, and Elvys Soares. 2026. Agentic LMs: Hunting Down Test Smells. IEEE Software 43, 1 (2026), 32–40.

Kate Moran and Kelly Gordon. 2023. How to Conduct a Heuristic Evaluation. [link]. Accessed: August 11, 2026.

Jakob Nielsen. 2024. 10 Usability Heuristics for User Interface Design. [link]. Originally published in 1994; updated January 30, 2024. Accessed: August 11, 2026.

Jakob Nielsen and Robert L. Mack. 1994. Usability Inspection Methods. John Wiley & Sons, New York.

Donald A. Norman. 2006. O design do dia a dia. Rocco, Rio de Janeiro.

Ali Ebrahimi Pourasad and Walid Maalej. 2025. Does GenAI Make Usability Testing Obsolete?. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). 437–449. DOI: 10.1109/ICSE55347.2025.00138

Raquel Oliveira Prates et al. 2003. Avaliação de interfaces de usuário–conceitos e métodos. In Jornada de Atualização em Informática do Congresso da Sociedade Brasileira de Computação, Capítulo, Vol. 6. sn, 28.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. Advances in neural information processing systems 30.

Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936.

Jason Wei et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.

Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Vol. 236. Springer, Berlin, Heidelberg. DOI: 10.1007/978-3-642-29044-2

Mingyuan Zhong and others. 2025. ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, Article 1163, 19 pages. DOI: 10.1145/3706598.3713797