Evaluating RAG Strategies in a Modular LLM Architecture

Vinicius Aguiar Alboneti; Leonardo Afonso Amorim; Jonatas Tomazini; Ricardo Costa; Sávio Salvarino Teles de Oliveira; Arlindo Rodrigues Galvão Filho; Anderson da Silva Soares; Tales Brumon Medeiros de Figueiredo

doi:10.5753/erigo.2025.17071

Vinicius Aguiar Alboneti UFG
Leonardo Afonso Amorim UFG
Jonatas Tomazini UFG
Ricardo Costa UFG
Sávio Salvarino Teles de Oliveira UFG
Arlindo Rodrigues Galvão Filho UFG
Anderson da Silva Soares UFG
Tales Brumon Medeiros de Figueiredo CEMIG

DOI: https://doi.org/10.5753/erigo.2025.17071

Resumo

This paper presents an empirical evaluation of three Retrieval-Augmented Generation (RAG) patterns — Naive RAG, RAG-Fusion, and Self-RAG — applied in a real-world corporate environment in the energy sector. Using a modular and consolidated architecture built on managed services and widely adopted technologies (such as pgvector, Cloud Run, and LLM APIs), we conducted experiments with actual data from an energy company, focusing on regulatory audit processes. We compared RAG strategies using automated evaluation metrics (RAGAS), considering faithfulness, contextual precision, answer relevance, and response time. The results show that the Self-RAG pattern achieves the best balance between response quality and performance, making it the most suitable for enterprise applications that require accuracy, efficiency, and scalability. This practical validation offers guidance on adopting RAG in corporate environments, highlighting the trade-offs associated with selecting different approaches.

Palavras-chave: Retrieval-Augmented Generation, RAG, LLM Architecture, Energy Sector, RAGAS

Referências

Albuquerque, A., Wensing, I., Filho, N. J., and Dorneles, C. (2024). Avaliação de aplicações de geração aumentada de recuperação por meio de feedback implícito. In Anais Estendidos do XXXIX Simpósio Brasileiro de Bancos de Dados, pages 253–259, Porto Alegre, RS, Brasil. SBC.

Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. (2024). Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint.

Es, S., James, J., Espinosa Anke, L., and Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Aletras, N. and De Clercq, O., editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Medeiros, A., Cavalcante, C., Nepomuceno, J., Lago, L., Ruberg, N., and Lifschitz, S. (2024). Contrato360: uma aplicação de perguntas e respostas usando modelos de linguagem, documentos e bancos de dados. In Anais do XXXIX Simpósio Brasileiro de Bancos de Dados, pages 155–166, Porto Alegre, RS, Brasil. SBC.

Qiao, Z., Gao, M., Ma, H., Wen, X., Yan, E., Zhao, W., Cao, H., and Wu, H. (2024). Intelligent question answering system for power regulations based on rag. In 2024 4th International Conference on New Energy and Power Engineering (ICNEPE), pages 1155–1161. IEEE.

Rackauckas, Z. (2024). Rag-fusion: a new take on retrieval-augmented generation. arXiv preprint.

Raihan, A. (2023). A comprehensive review of artificial intelligence and machine learning applications in the energy sector. Journal of Technology Innovations and Energy, 2:1–26.

Wang, H. and Iida, F. (2024). Emotional alignment for human-robot cooperation in musical tasks. IOP Conference Series: Materials Science and Engineering, 1321(1):012006.

Zhao, S., Yang, Y., Wang, Z., He, Z., Qiu, L. K., and Qiu, L. (2024). Retrieval augmented generation (rag) and beyond: A comprehensive survey on how to make your llms use external data more wisely. arXiv preprint arXiv:2409.14924.

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. (2025). A survey of large language models.