Application of Retrieval-Augmented Generation (RAG) in the Automated Extraction of Legal Arguments in Labor Administrative Defenses
Resumo
This paper investigates the use of Retrieval-Augmented Generation (RAG) for the automated extraction of legal arguments from statements of defense submitted in Brazilian labor-inspection proceedings (hereafter, administrative defenses). The main contribution is a transparent and auditable pipeline that combines semantic segmentation, vector retrieval, domain-specific legal embeddings, large language models, similarity-based validation, and human verification to improve traceability and reduce hallucinations. The system was evaluated on real defense documents and expert-provided ground truth using BERTScore, cosine similarity, and DeepEval-based qualitative criteria, with GPT-4o excluded from the DeepEval comparison to avoid judge-model bias. The results show that specialized legal embeddings improve argument identification, with STJ IRIS achieving the highest average cosine similarity among embeddings (0.850). In the full evaluation, the best BERTScore F1 score (0.871) was obtained by GPT-4o; when the evaluation was restricted to validated arguments, Llama 3.3 70B and Llama 4 Scout 17B reached an F1 score of 0.925. These findings indicate that RAG combined with legal embeddings and semantic validation can support reliable legal argument extraction.
Referências
Bruno, A., Mazzeo, P. L., Chetouani, A., Tliba, M., and Kerkouri, M. A. (2023). Insights into classifying and mitigating llms’ hallucinations.
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025). Current applications and challenges in large language models for patient care: a systematic review. Communications Medicine, 5(1):26.
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., and Xie, X. (2024). A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15(3).
Chen, Z. Z., Ma, J., Zhang, X., Hao, N., Yan, A., Nourbakhsh, A., Yang, X., McAuley, J., Petzold, L., and Wang, W. Y. (2024). A survey on large language models for critical societal domains: Finance, healthcare, and law.
Coelho, A. Z. (2024). A transformação digital e o uso de técnicas inteligência artificial (ia) no sistema de justiça do brasil. e-Publica, 11(1).
Juvekar, K. and Purwar, A. (2024). Cos-mix: Cosine similarity and distance fusion for improved information retrieval.
Lee, J., Stevens, N., and Han, S. C. (2025). Large language models in finance (finllms). Neural Computing and Applications, 37(30):24853–24867.
Louis, A., van Dijck, G., and Spanakis, G. (2024). Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. AAAI Press.
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., and Ho, D. E. (2025). Hallucination-free? assessing the reliability of leading ai legal research tools. Journal of Empirical Legal Studies, 22(2):216–242.
Melo, R., Santos, P. A., and Dias, J. (2023). A semantic search system for the supremo tribunal de justiça. In Moniz, N., Vale, Z., Cascalho, J., Silva, C., and Sebastião, R., editors, Progress in Artificial Intelligence, pages 142–154, Cham. Springer Nature Switzerland.
Rawte, V., Sheth, A., and Das, A. (2023). A survey of hallucination in large foundation models.
Shool, S., Adimi, S., Saboori Amleshi, R., Bitaraf, E., Golpira, R., and Tara, M. (2025). A systematic review of large language model (llm) evaluations in clinical medicine. BMC Medical Informatics and Decision Making, 25(1):117.
Wiratunga, N., Abeyratne, R., Jayawardena, L., Martin, K., Massie, S., Nkisi-Orji, I., Weerasinghe, R., Liret, A., and Fleisch, B. (2024). Cbr-rag: Case-based reasoning for retrieval augmented generation in llms for legal question answering. In Recio-Garcia, J. A., Orozco-del Castillo, M. G., and Bridge, D., editors, Case-Based Reasoning Research and Development, pages 445–460, Cham. Springer Nature Switzerland.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2020). Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
