A Comparative Analysis of LLMs in a FAQ Chatbot: A Case Study
Resumo
Este artigo apresenta um estudo experimental sobre o uso de modelos de linguagem de grande porte (LLMs) em um chatbot de FAQ. O sistema utiliza uma abordagem de geração aumentada por recuperação (RAG) e considera diferentes perfis de usuário. A avaliação combina métricas léxicas, semânticas e baseadas em LLM como juiz, além de indicadores operacionais como latência e custo. Os resultados mostram que métricas léxicas penalizam respostas mais detalhadas, enquanto métricas semânticas e baseadas em julgamento capturam melhor a qualidade das respostas. O estudo evidencia trade-offs entre desempenho, custo e adaptação aos perfis de usuário.
Referências
Attigeri, G., Agrawal, A., and Kolekar, S. V. (2024). Advanced nlp models for technical university information chatbots: development and comparative analysis. IEEE Access, 12:29633–29647.
Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang, B. (2024). Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327.
Fan, J., Xiao, W., Yan, Z., and Ouyang, Q. (2025). Accuracy of ai chatbots in answering frequently asked questions on cervical cancer. Frontiers in Artificial Intelligence, 8:1655303.
Foosherian, M., Purwins, H., Rathnayake, P., Alam, T., Teimao, R., and Thoben, K.-D. (2023). Enhancing pipeline-based conversational agents with large language models. In Proceedings of the 1st workshop on taming large language models: controllability in the era of interactive assistants!, pages 56–67.
Gomes, J. a. E. A., Prudêncio, R. B. C., and Nascimento, A. C. A. (2018). Chatbot to support frequently asked questions from students in higher education institutions. In Anais do XV Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), Porto Alegre, RS, Brasil. SBC.
Gomes, T., Regino, A. G., Caus, R., Sotelo, V., and dos Reis, J. C. (2025). Multi-agent llm approach for moderating e-commerce customer service responses. In Brazilian Symposium on Multimedia and the Web (WebMedia), pages 349–357. SBC.
Jiang, B., Hao, Z., Cho, Y.-M., Li, B., Yuan, Y., Chen, S., Ungar, L., Taylor, C. J., and Roth, D. (2025). Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale. arXiv preprint arXiv:2504.14225.
Kuratomi, G., Pirozelli, P., Cozman, F. G., and Peres, S. M. (2024). A RAG-Based institutional assistant. In Anais do XXI Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), Porto Alegre, RS, Brasil. SBC.
Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
Liu, C.-W., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., and Pineau, J. (2016). How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2122–2132.
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023. arXiv preprint arXiv:2303.16634, 12:1.
Luo, B., Lau, R. Y., Li, C., and Si, Y.-W. (2022). A critical review of state-of-the-art chatbot designs and applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1):e1434.
Narimissa, E. and Raithel, A. (2024). Exploring information retrieval landscapes: An investigation of a novel evaluation techniques and comparative document splitting methods. arXiv preprint arXiv:2409.08479.
Nusair, O., Asadigandomani, H., Osuorah, I., Sanchez, J., and Soleimani, M. (2025). Evaluating the reliability of large language models in answering faqs for cataract surgery. Digital Health, 11:20552076251406304.
Papadopoulos, T., Alexopoulos, C., and Charalabidis, Y. (2025). Evaluating chatbot architectures for public service delivery: balancing functionality, safety, ethics, and adaptability. Frontiers in Political Science, 7:1601440.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
Salim, M. S., Hossain, S. I., Jalal, T., Bose, D. K., and Basher, M. J. I. (2025). Llm based qa chatbot builder: A generative ai-based chatbot builder for question answering. SoftwareX, 29:102029.
Singh, S. U. and Namin, A. S. (2025). A survey on chatbots and large language models: Testing and evaluation techniques. Natural Language Processing Journal, 10:100128.
Soares, J. R. and Silva, P. N. (2025). Concepção de chatbots: perspectivas e interseções na ciência da informação. RDBCI: Revista Digital de Biblioteconomia e Ciência da Informação, 23:e025031.
Xu, F., Song, Y., Iyyer, M., and Choi, E. (2023). A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Zhao, Z., Vania, C., Kayal, S., Khan, N., Cohen, S. B., and Yilmaz, E. (2025). Personalens: A benchmark for personalization evaluation in conversational ai assistants. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18023–18055.
Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang, B. (2024). Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327.
Fan, J., Xiao, W., Yan, Z., and Ouyang, Q. (2025). Accuracy of ai chatbots in answering frequently asked questions on cervical cancer. Frontiers in Artificial Intelligence, 8:1655303.
Foosherian, M., Purwins, H., Rathnayake, P., Alam, T., Teimao, R., and Thoben, K.-D. (2023). Enhancing pipeline-based conversational agents with large language models. In Proceedings of the 1st workshop on taming large language models: controllability in the era of interactive assistants!, pages 56–67.
Gomes, J. a. E. A., Prudêncio, R. B. C., and Nascimento, A. C. A. (2018). Chatbot to support frequently asked questions from students in higher education institutions. In Anais do XV Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), Porto Alegre, RS, Brasil. SBC.
Gomes, T., Regino, A. G., Caus, R., Sotelo, V., and dos Reis, J. C. (2025). Multi-agent llm approach for moderating e-commerce customer service responses. In Brazilian Symposium on Multimedia and the Web (WebMedia), pages 349–357. SBC.
Jiang, B., Hao, Z., Cho, Y.-M., Li, B., Yuan, Y., Chen, S., Ungar, L., Taylor, C. J., and Roth, D. (2025). Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale. arXiv preprint arXiv:2504.14225.
Kuratomi, G., Pirozelli, P., Cozman, F. G., and Peres, S. M. (2024). A RAG-Based institutional assistant. In Anais do XXI Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), Porto Alegre, RS, Brasil. SBC.
Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
Liu, C.-W., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., and Pineau, J. (2016). How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 2122–2132.
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023. arXiv preprint arXiv:2303.16634, 12:1.
Luo, B., Lau, R. Y., Li, C., and Si, Y.-W. (2022). A critical review of state-of-the-art chatbot designs and applications. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(1):e1434.
Narimissa, E. and Raithel, A. (2024). Exploring information retrieval landscapes: An investigation of a novel evaluation techniques and comparative document splitting methods. arXiv preprint arXiv:2409.08479.
Nusair, O., Asadigandomani, H., Osuorah, I., Sanchez, J., and Soleimani, M. (2025). Evaluating the reliability of large language models in answering faqs for cataract surgery. Digital Health, 11:20552076251406304.
Papadopoulos, T., Alexopoulos, C., and Charalabidis, Y. (2025). Evaluating chatbot architectures for public service delivery: balancing functionality, safety, ethics, and adaptability. Frontiers in Political Science, 7:1601440.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
Salim, M. S., Hossain, S. I., Jalal, T., Bose, D. K., and Basher, M. J. I. (2025). Llm based qa chatbot builder: A generative ai-based chatbot builder for question answering. SoftwareX, 29:102029.
Singh, S. U. and Namin, A. S. (2025). A survey on chatbots and large language models: Testing and evaluation techniques. Natural Language Processing Journal, 10:100128.
Soares, J. R. and Silva, P. N. (2025). Concepção de chatbots: perspectivas e interseções na ciência da informação. RDBCI: Revista Digital de Biblioteconomia e Ciência da Informação, 23:e025031.
Xu, F., Song, Y., Iyyer, M., and Choi, E. (2023). A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Zhao, Z., Vania, C., Kayal, S., Khan, N., Cohen, S. B., and Yilmaz, E. (2025). Personalens: A benchmark for personalization evaluation in conversational ai assistants. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18023–18055.
Publicado
19/10/2026
Como Citar
XAVIER, Samara Monteiro; SANTOS, Cesar Yuri e Silva; CAVALCANTI, Anderson Pinheiro; MIRANDA, Péricles; MELLO, Rafael Ferreira.
A Comparative Analysis of LLMs in a FAQ Chatbot: A Case Study. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 23. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 2799-2810.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2026.30635.
