Small Language Models for Legal RAG in Portuguese: A 4×2 Cross-Domain Evaluation of Faithfulness, Retrieval Strategies, and Model Scale

• Guilherme Dutra UFG
• André Caraíba UFG
• Nádia Felix UFG
• Daniel Ribeiro UFG
• Állan Silva UFG
• Paulo Victor dos Santos UFG
• Pedro Albernaz UFG
• Sávio Teles UFG

DOI: https://doi.org/10.5753/eniac.2026.31444

Resumo

We present a 4×2 factorial evaluation of Retrieval-Augmented Generation (RAG) for Brazilian Portuguese legal text across four sub-domains of the JUÁ benchmark—Ulysses-RFCorpus (legislative), JurisTCU (judicial), BR-TaxQA (tax), and NormasTCU (regulatory). We compare BM25 Okapi and BGE-M3 retrieval across eight Small Language Models (SLMs, 1B–31B), with faithfulness scored by a Llama-3.3-70B judge against a Qwen-2.5-72B Oracle and validated via Cohen’s κ. Whereas prior work established that retrieval effectiveness is domain-dependent at the retrieval level, we provide the first end-to-end characterization of how this is reflected in generation faithfulness: BM25 dominates legislative, judicial, and regulatory text while dense retrieval wins only tax law, and 7–8B SLMs match models up to 31B with no detectable faithfulness difference. We further identify a truncation boundary where whole-document dense retrieval collapses on ultra-long norms, and a BERTScore Paradox that cautions Oracle-referenced evaluation. These results argue that domain-aware retrieval paired with 7–8B SLMs—rather than larger models—is the most defensible deployment path for Brazilian legal RAG. We report associations under fixed retrieval conditions and discuss the limits of a causal reading in Section 6.

Referências

Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., et al. (2024). Phi-4 technical report.

Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Muralidharan, S., Lin, Y. C., and Molchanov, P. (2025). Small language models are the future of agentic AI.

Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024). M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46.

Domingos Júnior, J., Faria, A., de Oliveira, E. S., de Brito, E., Teotonio, M., Assumpção, A., Carmo, D., Lotufo, R., and Pereira, J. (2025). BR-TaxQA-R: A dataset for question answering with references for Brazilian personal income tax law, including case law. In Intelligent Systems (BRACIS 2025). Springer.

Dubey, A. et al. (2024). The Llama 3 herd of models.

Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (EACL 2024), pages 150–158.

Fernandes, L. C., Ribeiro, L. d. S., Castro, M. V. B. d., Pacheco, L. A. d. S., and Sandes, E. F. d. O. (2025). JurisTCU: A Brazilian Portuguese information retrieval dataset with query relevance judgments. Language Resources and Evaluation.

Gemma Team, Google (2025). Gemma 3 technical report.

Gemma Team, Google (2026). Gemma 4 model card. [link].

Jiang, Z., Ma, X., and Chen, W. (2024). LongRAG: Enhancing retrieval-augmented generation with long-context LLMs.

Järvelin, K. and Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422–446.

Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. (2024). Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4334–4353.

Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174.

Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9459–9474.

Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81.

Liu, S., Li, X., Liu, Z., Yan, Y., Yang, C., Zeng, Z., Liu, Z., Sun, M., and Yu, G. (2025). Judge as a judge: Improving the evaluation of retrieval-augmented generation through the judge-consistency of large language models.

Pandey, S. et al. (2026). Can small language models use what they retrieve? an empirical study of retrieval utilization across model scale.

Pereira, J., Fernandes, L. C., de Brito, E., Lotufo, R., and Bonifacio, L. (2026). JUÁ – a benchmark for information retrieval in Brazilian legal text collections.

Qwen Team (2025). Qwen3 technical report.

Robertson, S. and Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389.

Rosenthal, S., Shah, V., Katsis, Y., and Danilevsky, M. (2026). SemEval-2026 task 8 (MTRAGEval): Evaluating multi-turn RAG conversations. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).

Salemi, A. and Zamani, H. (2024). Evaluating retrieval quality in retrieval-augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Shi, L., Ma, C., Liang, W., Diao, X., Ma, W., and Vosoughi, S. (2025). Judging the judges: A systematic study of position bias in LLM-as-a-judge. In Proceedings of IJCNLP-AACL 2025.

Souza, E., Vitório, D., Moriyama, G., Santos, L., Martins, L., Souza, M., Fonseca, M., Félix, N., Carvalho, A., Albuquerque, H. O., and Oliveira, A. L. I. (2021). An information retrieval pipeline for legislative documents from the Brazilian Chamber of Deputies. In Legal Knowledge and Information Systems (JURIX 2021), pages 119–126.

Vitório, D., Souza, E., Martins, L., da Silva, N. F. F., de Carvalho, A. C. P. d. L., Oliveira, A. L. I., and de Andrade, F. E. (2025). Building a relevance feedback corpus for legal information retrieval in the real-case scenario of the Brazilian Chamber of Deputies. Language Resources and Evaluation, 59:1257–1277.

Zhang, Q., Wang, Y., Yu, T., Jiang, Y., Wu, C., Li, L., Wang, Y., Jiang, X., Shang, L., Tang, R., Lyu, F., and Ma, C. (2024). RevisEval: Improving LLM-as-a-Judge via response-adapted references.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. In Proceedings of ICLR 2020.