A Multidimensional LLM Evaluation Pipeline for Automatic Responses to Restaurant Reviews in Brazilian Portuguese
Resumo
Este trabalho avalia GPT-5.5, Gemini 3.1 e Llama 4 na geração automática de respostas a avaliações de restaurantes em português brasileiro, tarefa que exige empatia, adequação de tom, fluência e consistência de marca. Um experimento fatorial (3× 2× 2× 2 = 2.400 respostas) combina 15 avaliadores humanos (Likert 1–5; concordância robusta nas dimensões pragmáticas, Fleiss’ κ ∈ [0,76; 0,81], e moderada em fluência, κ = 0,48) com quatro métricas automáticas. Nenhum modelo é universalmente superior: Llama 4 lidera em empatia, Gemini 3.1 em adequação de tom e consistência de marca, e GPT-5.5 em fluência. Não há evidência de superioridade consistente da estratégia zero-shot sobre a few-shot nas dimensões pragmáticas (p = 0,47 em empatia). Mais relevante, há discordância de ranking entre as métricas automáticas e o julgamento humano nas dimensões pragmáticas. Em adequação de tom, o modelo mais bem avaliado pelos humanos é o pior pelo proxy automático, o que desaconselha seu uso isolado em tarefas com nuances pragmáticas.
Referências
Abbasi, A. and Chen, H. (2008). Writeprints: A stylometric approach to identity-level identification and similarity detection in cyberspace. ACM Transactions on Information Systems (TOIS), 26(2):7.
Barriere, V., Sedoc, J., Tafreshi, S., and Giorgi, S. (2023). Findings of wassa 2023 shared task on empathy, emotion and personality detection in conversation and reactions to news articles. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 511–525.
Barriere, V., Tafreshi, S., Sedoc, J., and Alqahtani, S. (2022). Wassa 2022 shared task: Predicting empathy, emotion and personality in reaction to news stories. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 214–227.
Brown, T., Mann, B., Ryder, N., Subbiah, M., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901.
Crolic, C., Thomaz, F., Hadi, R., and Stephen, A. T. (2022). Blame the bot: Anthropomorphism and anger in customer–chatbot interactions. Journal of Marketing, 86(1):132–148.
Danescu-Niculescu-Mizil, C., Sudhof, M., Jurafsky, D., Leskovec, J., and Potts, C. (2013). A computational approach to politeness with application to social factors. In Proceedings of the 51st Annual Meeting of the ACL, pages 250–259.
de Araújo, G., de Melo, T., and Figueiredo, C. M. S. (2024). Explorando a eficácia das linguagens generativas em tarefas de análise de sentimentos no português brasileiro. Linguamática, 16(2):41–58.
Howcroft, D. M. and Rieser, V. (2021). What happens if you treat ordinal ratings as interval data? human evaluations in nlp are even more under-powered than you think. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8932–8939.
Hudeček, V. and Dušek, O. (2023). Are large language models all you need for task-oriented dialogue? In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 216–228.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, W. B. (2016). A diversity-promoting objective function for neural conversation models. In Proceedings of NAACL-HLT, pages 110–119.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.
Mello, R. F., Rodrigues, L., Cabral, L., Pereira, F. D., et al. (2024). Prompt engineering for automatic short answer grading in brazilian portuguese. In SBIE, pages 1730–1743. SBC.
Pistotti, T., Witbrock, M., O’Leary, P. A. T., and Brown, J. (2025). The benefits of being uncertain: Perplexity as a signal for naturalness in multilingual machine translation. In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 61–65.
Rohden, S. F. and Espartel, L. B. (2026). Emotional artificial intelligence: The impact of chatbot empathy and emotional tone on consumer satisfaction and word of mouth. International Journal of Human-Computer Studies, 210:103764.
Sharma, A., Miner, A. S., Atkins, D. C., and Althoff, T. (2020). A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276.
Silva, L. A., Biancolino, C. A., and Spers, E. E. (2019). The effect of online reviews and comments on the intention of visiting restaurants: An experimental study. Anais do CLAV 2019 - 19º Congresso Latino-Americano de Varejo, 12:1–15.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS).
Barriere, V., Sedoc, J., Tafreshi, S., and Giorgi, S. (2023). Findings of wassa 2023 shared task on empathy, emotion and personality detection in conversation and reactions to news articles. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 511–525.
Barriere, V., Tafreshi, S., Sedoc, J., and Alqahtani, S. (2022). Wassa 2022 shared task: Predicting empathy, emotion and personality in reaction to news stories. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 214–227.
Brown, T., Mann, B., Ryder, N., Subbiah, M., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901.
Crolic, C., Thomaz, F., Hadi, R., and Stephen, A. T. (2022). Blame the bot: Anthropomorphism and anger in customer–chatbot interactions. Journal of Marketing, 86(1):132–148.
Danescu-Niculescu-Mizil, C., Sudhof, M., Jurafsky, D., Leskovec, J., and Potts, C. (2013). A computational approach to politeness with application to social factors. In Proceedings of the 51st Annual Meeting of the ACL, pages 250–259.
de Araújo, G., de Melo, T., and Figueiredo, C. M. S. (2024). Explorando a eficácia das linguagens generativas em tarefas de análise de sentimentos no português brasileiro. Linguamática, 16(2):41–58.
Howcroft, D. M. and Rieser, V. (2021). What happens if you treat ordinal ratings as interval data? human evaluations in nlp are even more under-powered than you think. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8932–8939.
Hudeček, V. and Dušek, O. (2023). Are large language models all you need for task-oriented dialogue? In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 216–228.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, W. B. (2016). A diversity-promoting objective function for neural conversation models. In Proceedings of NAACL-HLT, pages 110–119.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.
Mello, R. F., Rodrigues, L., Cabral, L., Pereira, F. D., et al. (2024). Prompt engineering for automatic short answer grading in brazilian portuguese. In SBIE, pages 1730–1743. SBC.
Pistotti, T., Witbrock, M., O’Leary, P. A. T., and Brown, J. (2025). The benefits of being uncertain: Perplexity as a signal for naturalness in multilingual machine translation. In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), pages 61–65.
Rohden, S. F. and Espartel, L. B. (2026). Emotional artificial intelligence: The impact of chatbot empathy and emotional tone on consumer satisfaction and word of mouth. International Journal of Human-Computer Studies, 210:103764.
Sharma, A., Miner, A. S., Atkins, D. C., and Althoff, T. (2020). A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276.
Silva, L. A., Biancolino, C. A., and Spers, E. E. (2019). The effect of online reviews and comments on the intention of visiting restaurants: An experimental study. Anais do CLAV 2019 - 19º Congresso Latino-Americano de Varejo, 12:1–15.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS).
Publicado
19/10/2026
Como Citar
BRUNO, Danilo; FIGUEIREDO, Carlos M. S.; PONTES, Danielle; MELO, Tiago de.
A Multidimensional LLM Evaluation Pipeline for Automatic Responses to Restaurant Reviews in Brazilian Portuguese. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 23. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 2503-2514.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2026.29835.
