Evaluation of the Impact of Different RAG Architectural Patterns in Legal Domains

• Salvador Ludovico Paranhos UFG
• Jonatas Novais Tomazini UFG
• Celso Goncalves Camilo Junior UFG
• Savio Salvarino Teles de Oliveira UFG

DOI: https://doi.org/10.5753/erigo.2024.4846

Abstract

This study evaluates the impact of different Retrieval-Augmented Generation (RAG) architectures in the legal context, focusing on the accuracy and relevance of responses in question-answering (Q&A) systems. Variations in query manipulation strategies, document retrieval, and relevance checks were investigated to analyze how these factors influence the quality of responses for legal queries. Multiple RAG architectures were implemented, along with a synthesizer module and an evaluator module to compare the efficiency of these patterns. The results indicate that RAG architecture performance varies significantly depending on the type of legal query, highlighting that understanding these dynamics is essential for optimizing Q&A systems within the legal domain.

Keywords: RAG, architectures, legal domain, Q&A systems, information retrieval

References

Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. (2023). Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv.

Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2023). Ragas: Automated evaluation of retrieval augmented generation. arXiv.

Fan, Wenqi, Ding, Yujuan, Ning, Liangbo, Wang, Shijie, Li, Hengyun, Yin, Dawei, Chua, Tat-Seng, Li, and Qing (2024). A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501.

Gao, L., Ma, X., Lin, J., and Callan, J. (2022). Hprecise zero-shot dense retrieval without relevance labels. arXiv.

Jagerman, R., Zhuang, H., Qin, Z., Wang, X., and Bendersky, M. (2023). Query expansion by prompting large language models. arXiv.

Krasadakis, Panteleimon, Sakkopoulos, Evangelos, Verykios, and S, V. (2024). A survey on challenges and advances in natural language processing with a focus on legal informatics and low-resource languages. Electronics, 13(3):648.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv.

Ma, X., Gong, Y., He, P., Zhao, H., and Duan, N. (2023). Query rewriting for retrieval-augmented large language models. arXiv.

Rackauckas, Z. (2024). Rag-fusion: a new take on retrieval-augmented generation. arXiv.

Saad-Falcon, Jon, Khattab, Omar, Potts, Christopher, Zaharia, and Matei (2023). ARES: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09476.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv.

Wiratunga, Nirmalie, Abeyratne, Ramitha, Jayawardena, Lasal, Martin, Kyle, Massie, Stewart, Nkisi-Orji, Ikechukwu, Weerasinghe, Ruvan, Liret, Anne, Fleisch, and Bruno (2024). Cbr-rag: Case-based reasoning for retrieval augmented generation in llms for legal question answering. In International Conference on Case-Based Reasoning, pages 445–460. Springer.

Yan, S.-Q., Gu, J.-C., Zhu, Y., and Ling, Z.-H. (2024). Corrective retrieval augmented generation. arXiv.

Zhou, P., Pujara, J., Ren, X., Chen, X., Cheng, H.-T., Le, Q. V., Chi, E. H., Zhou, D., Mishra, S., and Zheng, H. S. (2024). Self-discover: Large language models self-compose reasoning structures. arXiv preprint arXiv:2402.03620.