Evaluation of RAG Retrieval Strategies with Quantized Qwen3 Models for University Documents
Resumo
Este estudo comparou cinco estratégias de recuperação para sistemas locais de Geração Aumentada por Recuperação (RAG), utilizando três modelos Qwen3 quantizados, 37 documentos institucionais, 50 perguntas e três repetições por configuração. A estratégia reranking obteve as maiores medianas agregadas de Context Precision e Answer Correctness. O Qwen3 8B com reranking alcançou o melhor resultado individual. Embora naive e hybrid apresentassem menor latência, o reranking proporcionou o melhor equilíbrio entre qualidade e latência no corpus avaliado.
Referências
Bruch, S., Gai, S., and Ingber, A. (2023). An analysis of fusion functions for hybrid retrieval. ACM Trans. Inf. Syst., 42(1).
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024). M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand. Association for Computational Linguistics.
Cormack, G. V., Clarke, C. L. A., and Buettcher, S. (2009). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 758–759, New York, NY, USA. Association for Computing Machinery.
Es, S., James, J., Espinosa Anke, L., and Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Aletras, N. and De Clercq, O., editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics.
Gao, L., Ma, X., Lin, J., and Callan, J. (2023). Precise zero-shot dense retrieval without relevance labels. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational Linguistics.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2024). Retrieval-augmented generation for large language models: A survey.
Jeong, S., Baek, J., Cho, S., Hwang, S. J., and Park, J. (2024). Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Duh, K., Gomez, H., and Bethard, S., editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050, Mexico City, Mexico. Association for Computational Linguistics.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. (2023). Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
Kulkarni, A., Jhaveri, P. R., Shrestha, P., Han, Y. T., Amini, R., and Madahian, B. (2026). Long-context long-form question answering for legal domain. In Matusevych, Y., Eryiğit, G., and Aletras, N., editors, Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), pages 728–751, Rabat, Morocco. Association for Computational Linguistics.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.
Nogueira, R. and Cho, K. (2019). Passage re-ranking with BERT. CoRR, abs/1901.04085.
Oliveira, V. T. d., Silva, D. O. d., Souza, M. d. A., Lima, M. R., Oliveira, S. S. T. d., and Rosa, T. C. (2026). Retrieval-augmented generation and knowledge graphs in Portuguese-language legal documents. In Souza, M., de Dios-Flores, I., Santos, D., Freitas, L., Souza, J. W. d. C., and Ribeiro, E., editors, Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1, pages 1–10, Salvador, Brazil. Association for Computational Linguistics.
RAGAS (2026). List of available metrics. Available in: [link]. Accessed: 2026-05-20.
Shanahan, M. (2024). Talking about large language models. Commun. ACM, 67(2):68–79.
Van Opijnen, M. and Santos, C. (2017). On the concept of relevance in legal information retrieval. Artif. Intell. Law, 25(1):65–87.
Wojtasik, K., Berdowski, A., Okulska, I., and Piasecki, M. (2025). Polichat: Retrieval augmented generation on university documents and regulations. In Paszynski, M., Barnard, A. S., and Zhang, Y. J., editors, Computational Science – ICCS 2025 Workshops, pages 273–288, Cham. Springer Nature Switzerland.
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024). M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand. Association for Computational Linguistics.
Cormack, G. V., Clarke, C. L. A., and Buettcher, S. (2009). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 758–759, New York, NY, USA. Association for Computing Machinery.
Es, S., James, J., Espinosa Anke, L., and Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Aletras, N. and De Clercq, O., editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics.
Gao, L., Ma, X., Lin, J., and Callan, J. (2023). Precise zero-shot dense retrieval without relevance labels. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational Linguistics.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2024). Retrieval-augmented generation for large language models: A survey.
Jeong, S., Baek, J., Cho, S., Hwang, S. J., and Park, J. (2024). Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Duh, K., Gomez, H., and Bethard, S., editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7036–7050, Mexico City, Mexico. Association for Computational Linguistics.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. (2023). Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
Kulkarni, A., Jhaveri, P. R., Shrestha, P., Han, Y. T., Amini, R., and Madahian, B. (2026). Long-context long-form question answering for legal domain. In Matusevych, Y., Eryiğit, G., and Aletras, N., editors, Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), pages 728–751, Rabat, Morocco. Association for Computational Linguistics.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.
Nogueira, R. and Cho, K. (2019). Passage re-ranking with BERT. CoRR, abs/1901.04085.
Oliveira, V. T. d., Silva, D. O. d., Souza, M. d. A., Lima, M. R., Oliveira, S. S. T. d., and Rosa, T. C. (2026). Retrieval-augmented generation and knowledge graphs in Portuguese-language legal documents. In Souza, M., de Dios-Flores, I., Santos, D., Freitas, L., Souza, J. W. d. C., and Ribeiro, E., editors, Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1, pages 1–10, Salvador, Brazil. Association for Computational Linguistics.
RAGAS (2026). List of available metrics. Available in: [link]. Accessed: 2026-05-20.
Shanahan, M. (2024). Talking about large language models. Commun. ACM, 67(2):68–79.
Van Opijnen, M. and Santos, C. (2017). On the concept of relevance in legal information retrieval. Artif. Intell. Law, 25(1):65–87.
Wojtasik, K., Berdowski, A., Okulska, I., and Piasecki, M. (2025). Polichat: Retrieval augmented generation on university documents and regulations. In Paszynski, M., Barnard, A. S., and Zhang, Y. J., editors, Computational Science – ICCS 2025 Workshops, pages 273–288, Cham. Springer Nature Switzerland.
Publicado
19/10/2026
Como Citar
OLIVEIRA, Lucas S. de; GUARIENTI, César; JESUS, Leandro D. de; OLIVEIRA, Anderson C. S. de; MARTINS, Claudia A..
Evaluation of RAG Retrieval Strategies with Quantized Qwen3 Models for University Documents. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 23. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 1664-1674.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2026.31281.
