Critic-Based Self-Correction in Local Retrieval-Augmented Generation: An Empirical Evaluation in an Institutional Academic Chatbot

Gabriel de Souza Lima; Júlio César Mansano; Fernanda Diniz Marabin; João Emmanuel D’Alkmin Neves

doi:10.5753/eniac.2026.31037

Gabriel de Souza Lima FATEC
Júlio César Mansano FATEC
Fernanda Diniz Marabin FATEC
João Emmanuel D’Alkmin Neves FATEC

DOI: https://doi.org/10.5753/eniac.2026.31037

Resumo

This paper evaluates critic-based self-correction in local Retrieval-Augmented Generation for an institutional academic chatbot. Five configurations were compared: a baseline without critic, a deterministic critic, two critic variants with a Large Language Model judge, and a conditional oracle policy. Evaluation used RAGAS, end-to-end latency, and paired Wilcoxon tests by query type. The central finding is that critic-based self-correction reduces Answer Relevancy, our proxy for response utility, in constrained local environments: uniform critic activation increases latency substantially and does not compensate for the synthesis limitations of small-scale models. The conditional policy improves the quality-cost trade-off over always-on critics, but does not outperform the baseline globally.

Referências

Asai, A., Wu, Z., Wang, Y., Sil, A. and Hajishirzi, H. (2024). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In Proceedings of the International Conference on Learning Representations.

Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112(1), 155-159.

Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N. and Silvestri, F. (2024). The Power of Noise: Redefining Retrieval for RAG Systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Es, S., James, J., Anke, L. E. and Schockaert, S. (2024). RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150-158.

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J. and Wang, H. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997.

Jeong, S., Baek, J., Cho, S., Hwang, S. J. and Park, J. C. (2024). Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. In Proceedings of the 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics.

Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Yu, J. D., Yang, Y., Callan, J. and Neubig, G. (2023). Active Retrieval Augmented Generation. arXiv:2305.06983.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W.-t., Rocktaschel, T., Riedel, S. and Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33.

Neves, J. E. D. A.; Pedro, P. S. M.; Hernandez, M. F. G.; Fabri Junior, L. A. Simulation of the Implementation of Domestic Solar Systems Using Multi-agent Systems from Web Scraping. Smart Innovation, Systems and Technologies. 1ed.: Springer International Publishing, 2023, v. 1, p. 88-96.

Yan, S.-Q., Gu, J.-C., Zhu, Y. and Ling, Z.-H. (2024). Corrective Retrieval Augmented Generation. arXiv:2401.15884.

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E. and Stoica, I. (2024). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 37.