Uso de grandes modelos de linguagem na análise da experiência do usuário

• Yvens Almeida Girão UFC
• Denise da Silva Santos UFC
• Pedro Oliveira Unicamp
• Marília Mendes UFC
• Alexandre Arruda UFC

DOI: https://doi.org/10.5753/webmedia.2026.30196

Resumo

This work investigates the application of Large Language Models (LLMs) to the classification stage of the MALTU method for analyzing Postings Related to Use (PRUs). Using 299 reviews extracted from the Gov.br app page on Google Play, annotated by human evaluators as a gold standard, we evaluated five LLMs (GPT-4o, GPT-4o-mini, GPT-OSS-120B, Kimi K2, and Sabiazinho-4) under two prompting strategies (zero-shot and few-shot). Models were instructed to return structured JSON outputs for the six MALTU dimensions: type, intention, sentiment, functionality, usability, and UX. In total, models produced 2,990 classifications that were compared against the human gold standard using accuracy, precision, recall, and macro F1. Average accuracy ranged approximately from 74% to 78% (Kimi K2 achieved the best performance in the few-shot setting: 77.8%; GPT-OSS-120B led the zero-shot setting: 76.8%). Vocabulary dimensions (intention, type, sentiment, and UX) exhibited high performance (∼85–95%), whereas functionality (33–49%) and usability (57–65%) were more challenging due to terminological variation and semantic ambiguity.

Palavras-chave: Large Language Models, MALTU, automated classification, user experience, prompt engineering

Referências

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877–1901.

Peitong Duan, JeremyWarner, and Björn Hartmann. 2023. Towards Generating UI Design Feedback with LLMs. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings (UIST ’23 Adjunct). Association for Computing Machinery, New York, NY, USA, 1–3. DOI: 10.1145/3586182.3615810

Emanuel Felipe Duarte, Paula Toledo Palomino, Taciana Pontual Falcão, Grace Lis Proença Meireles Barreto Porto, Carlos dos Santos Portela, Douglas Francisco Ribeiro, André Nascimento, Yuska Paola Costa Aguiar, Maurício Souza, Angelita Moutin Segoria Gasparotto, and Armando Maciel Toda. 2024. GranDIHC-BR 2025–2035 - GC6: Implications of Artificial Intelligence in HCI: A Discussion on Paradigms, Ethics, and Diversity, Equity and Inclusion. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems (IHC ’24). ACM, Article 22, 19 pages. DOI: 10.1145/3702038.3702059

Lúcio França, Lisieux Andrade, and Windson Carvalho. 2025. Um Estudo Sobre o Uso de Prompts LLM para a Avaliação da Acessibilidade Web dos Portais das Prefeituras da Região dos Sertões dos Crateús. In Proceedings of the 31st Brazilian Symposium on Multimedia and the Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 580–584. DOI: 10.5753/webmedia.2025.16064

Andrew Kean Gao. 2023. Prompt Engineering for Large Language Models: A brief guide with examples for non-technical readers. Stanford University Tech Report / Online Guide. Acedido em: Maio de 2026.

Pawanjit Singh Ghatora, Seyed Ebrahim Hosseini, Shahbaz Pervez, Muhammad J. Iqbal, and Nabil Shaukat. 2024. Sentiment Analysis of Product Reviews Using Machine Learning and Pre-Trained LLM. Big Data Cogn. Comput. 8 (2024), 199. DOI: 10.3390/bdcc8120199

Google Play Console Help. 2024. Avaliações, notas e métricas de apps. [link]. Acessado em: 23 de maio de 2024.

Google Research. 2017. Google Colaboratory. [link]. Acesso em: 13 maio 2026.

Alexandra-Elena Gurita and Radu-Daniel Vatavu. 2025. When LLM-Generated Code Perpetuates User Interface Accessibility Barriers, How Can We Break the Cycle?. In Proceedings of the 22nd International Web for All Conference (W4A ’25) (Sydney, NSW, Australia). ACM, New York, NY, USA. DOI: 10.1145/3744257.3744266

Nien-Lin Hsueh, Hsuen-Jen Lin, and Lien-Chi Lai. 2024. Applying Large Language Model to User Experience Testing. Electronics 13, 23 (24 Nov 2024), 4633. DOI: 10. 3390/electronics13234633 Open access; published 24 November 2024.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems 35 (2022), 22199–22213.

Kristian Kolthoff, Felix Kretzer, Lennart Fiebig, Christian Bartelt, Alexander Maedche, and Simone Paolo Ponzetto. 2024. Zero-shot prompting approaches for llm-based graphical user interface generation. arXiv preprint arXiv:2412.11328 (2024).

Yoonjoo Lee, Hyunjin Kim, Minji Park, and Juho Kim. 2025. Leveraging Multimodal LLM for Inspirational User Interface Search. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/3706598.3714123

LMSYS Org. 2026. Chatbot Arena Leaderboard. [link] Acedido em: Maio de 2026.

Marília Soares Mendes. 2015. MALTU – um modelo para avaliação da interação em sistemas sociais a partir da linguagem textual do usuário. Ph.D. Dissertation. Universidade Federal do Ceará, Fortaleza, Brazil. Advisor(s) Castro, Miguel Franklin de. Tese (Doutorado em Ciência da Computação).

n8n GmbH. 2026. n8n: Workflow Automation Tool. [link] Acedido em: 29 de maio de 2026.

Donald A. Norman. 2004. Emotional Design: Why We Love (or Hate) Everyday Things. Basic Books, New York, NY.

George Oliveira and Simone Barbosa. 2025. Evaluating ChatGPT to support data visualization design. In Anais do XXIV Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (Belo Horizonte/MG). SBC, Porto Alegre, RS, Brasil, 779–797. DOI: 10.5753/ihc.2025.11894

Hismael Oliveira, Alexandre Arruda, and Marilia Mendes. 2020. Identificação Automática de Postagens Relacionadas ao Uso Através de Modelos de Aprendizagem Profunda. In Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC). Salvador, 587. XVI Encontro Nacional de Inteligência Artificial e Computacional.

OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023).

OpenAI. 2025. Introducing GPT-OSS. [link] Acedido em: 29 de maio de 2026.

Sinan Ozdemir. 2023. Quick Start Guide to Large Language Models: Strategies and Best Practices for ChatGPT, Embeddings, Fine-Tuning, and Multimodal AI. Addison-Wesley Professional.

Rock Yuren Pang, Hope Schroeder, Kynnedy Simone Smith, Solon Barocas, Ziang Xiao, Emily Tseng, and Danielle Bragg. 2025. Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/3706598.3713726

Daniel Mesquita Feijó Rabelo, Ribamar de Souza Martins, Isaac Santos, Paulo Henrique G. Da Silva, Kiev Gama, and Windson Viana. 2025. Breaking Barriers in Mobile Accessibility: A Study of LLM-Generated Native Android Interfaces. In 2025 IEEE/ACM 12th International Conference on Mobile Software Engineering and Systems (MOBILESoft). 20–31. DOI: 10.1109/MOBILESoft66462.2025.00010

Mohaimenul Azam Khan Raiaan, Md. Saddam Hossain Mukta, Kaniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most. Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eunus Ali, and Sami Azam. 2024. A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges. IEEE Access 12 (2024), 26839–26874. DOI: 10.1109/ACCESS.2024.3365742

Serviços e Informações do Brasil. 2026. gov.br – Apps no Google Play. [link]. Acesso em: 21 ago. 2026.

Cole Stryker. [n. d.]. O que é LLM (grandes modelos de linguagem)? [link]. Acedido em: 29 de maio de 2026.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems, Vol. 30. [link]

Carlos Ventura, Daniel Rabelo, Ticianne Darin, and Windson Viana. 2025. Accessibility Evaluation of LLM-Generated Android Code in React Native. In Anais do XXIV Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (Belo Horizonte/MG). SBC, Porto Alegre, RS, Brasil, 1301–1322. DOI: 10.5753/ihc.2025.10833

Xingyu Wu, Sheng-Hao Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. 2025. Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap. IEEE Transactions on Evolutionary Computation 29, 2 (2025), 534–554. DOI: 10.1109/TEVC.2024.3506731

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).