Retrieval-Augmented Generation for Question Answering over Brazilian Portuguese Institutional Normative Documents

• Avelar Rodrigues de Sousa USP
• Jean Carlos Rodrigues Sousa UFPI
• Karielly de Carvalho UFPI
• Manoel Messias Pereira Medeiros IFPI
• Marx Rodrigues de Moura IFPI

DOI: https://doi.org/10.5753/eniac.2026.31559

Resumo

This paper evaluates a Retrieval-Augmented Generation (RAG) prototype for question answering over Brazilian Portuguese institutional normative documents. The study uses an IFPI document subset, 20 factual questions, 10 independent runs, four retrieval settings, and two generator scales: gemma3:12b and gemma3:27b. Results show that dense retrieval outperformed lexical baselines in end-to-end answer quality, while reranking improved retrieval metrics without necessarily improving generated answers. The larger generator achieved modest semantic gains but was substantially slower. Error analysis showed limitations in metadata-sensitive questions involving document numbers, dates, and administrative references. The findings provide exploratory evidence about standard RAG components in an institutional public-information scenario and highlight the need for explicit source grounding and exact normative verification.

Referências

Albuquerque, A., Wensing, I., Joppi Filho, N., and Dorneles, C. (2024). Avaliação de aplicações de geração aumentada de recuperação por meio de feedback implícito. In Anais Estendidos do XXXIX Simpósio Brasileiro de Bancos de Dados, pages 253–259, Porto Alegre, RS, Brasil. SBC.

Aquino, I., dos Santos, M. M., Dorneles, C., and Carvalho, J. T. (2024). Extracting information from brazilian legal documents with retrieval augmented generation. In Anais Estendidos do XXXIX Simpósio Brasileiro de Bancos de Dados, pages 280–287, Porto Alegre, RS, Brasil. SBC.

Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., and Li, Q. (2024). A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501, New York, NY, USA. Association for Computing Machinery.

Gan, A., Yu, H., Zhang, K., Liu, Q., Yan, W., Huang, Z., Tong, S., and Hu, G. (2025). Retrieval augmented generation evaluation in the era of large language models: A comprehensive survey. arXiv preprint arXiv:2504.14891.

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2024). Retrieval-augmented generation for large language models: A survey.

Huang, Y., Xu, Y. E., Sun, K., Yan, V., Sun, A., Khan, H., Nguyen, J., Chen, J., Kachuee, M., Lin, Z., Liu, Y., Colak, A., Kumar, A., Yih, W.-t., and Dong, X. L. (2025). Confrag: Confidence-guided retrieval-augmenting generation.

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.

Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6769–6781, Online. Association for Computational Linguistics.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474.

Novais, A., Ferreira, D., Silva, J., Brakes, M., Presa, J., and Oliveira, S. (2025). Enhancing legal question answering in brazilian portuguese through domain-specific embedding models. In Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional, pages 523–533, Porto Alegre, RS, Brasil. SBC.

Oliveira, V. T. d., Silva, D. O. d., Souza, M. d. A., Lima, M. R., Oliveira, S. S. T. d., and Rosa, T. C. (2026). Retrieval-augmented generation and knowledge graphs in Portuguese-language legal documents. In Souza, M., de Dios-Flores, I., Santos, D., Freitas, L., Souza, J. W. d. C., and Ribeiro, E., editors, Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1, pages 1–10, Salvador, Brazil. Association for Computational Linguistics.

Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. (2021). Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.

UNESCO (2021). To recovery and beyond: 2021 report on public access to information (sdg 16.10.2). Technical report, UNESCO, Paris.

Yoran, O., Wolfson, T., Ram, O., and Berant, J. (2024). Making retrieval-augmented language models robust to irrelevant context.