J-KGRAG: A Hybrid Retrieval-Augmented Generation Architecture for Legal Norm Understanding with Knowledge Graphs
Resumo
This paper presents Juridic KGRAG (J-KGRAG), a hybrid architecture that enhances Retrieval-Augmented Generation (RAG) by integrating structured legal knowledge through a domain-specific knowledge graph. The system is designed to address the challenge of retrieving up-to-date legal information in highly interdependent normative documents, a frequent scenario in the Brazilian public sector. The method is applied to a corpus of 42 normative acts from Court of Accounts of the State of Goiás, Brazil, where legal articles are frequently updated, repealed, or referenced by newer documents. J-KGRAG enriches standard dense retrieval with a graph-based expansion step that identifies and retrieves updated entities omitted in the initial search. Experimental results indicate a significant improvement in factual accuracy (+75%) and overall answer correctness (+16%) compared to a naive RAG baseline. In addition, a manually curated benchmark of 53 legal question–answer pairs is released, and a qualitative analysis is performed to highlight the advantages of structured retrieval. The results demonstrate that combining symbolic legal representations with LLM-based generation improves both the consistency and the reliability of answers in legal domains.
Palavras-chave:
Legal Question Answering, Knowledge Graphs, Retrieval-Augmented Generation (RAG)
Referências
S. T. Federal, “Justiça em números: presidente do stf divulga dados do judiciário brasileiro,” 2024, accessed: 2025-02-26. [Online]. Available: [link]
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrievalaugmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020.
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024.
X. He, Y. Tian, Y. Sun, N. V. Chawla, T. Laurent, Y. LeCun, X. Bresson, and B. Hooi, “G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,” arXiv preprint arXiv:2402.07630, 2024.
W. Fan, Y. Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6491–6501.
Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, 2023.
Y. Huang and J. Huang, “A survey on retrieval-augmented text generation for large language models,” arXiv preprint arXiv:2404.10981, 2024.
S. Wu, Y. Xiong, Y. Cui, H. Wu, C. Chen, Y. Yuan, L. Huang, X. Liu, T.-W. Kuo, N. Guan et al., “Retrieval-augmented generation for natural language processing: A survey,” arXiv preprint arXiv:2407.13193, 2024.
P. Zhao, H. Zhang, Q. Yu, Z. Wang, Y. Geng, F. Fu, L. Yang, W. Zhang, and B. Cui, “Retrieval-augmented generation for ai-generated content: A survey,” arXiv preprint arXiv:2402.19473, 2024.
H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of retrieval-augmented generation: A survey,” arXiv preprint arXiv:2405.07437, 2024.
M. Yasunaga, H. Ren, A. Bosselut, P. Liang, and J. Leskovec, “Qagnn: Reasoning with language models and knowledge graphs for question answering,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 535–546.
D. Taunk, L. Khanna, S. V. P. K. Kandru, V. Varma, C. Sharma, and M. Tapaswi, “Grapeqa: Graph augmentation and pruning to enhance question-answering,” in Companion Proceedings of the ACM Web Conference 2023, 2023, pp. 1138–1144.
S. Li, Y. Gao, H. Jiang, Q. Yin, Z. Li, X. Yan, C. Zhang, and B. Yin, “Graph reasoning for question answering with triplet retrieval,” arXiv preprint arXiv:2305.18742, 2023.
Y. Huang, Y. Li, Y. Xu, L. Zhang, R. Gan, J. Zhang, and L. Wang, “Mvp-tuning: Multi-view knowledge retrieval with prompt tuning for commonsense reasoning,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 13 417–13 432.
Y. Hu, Z. Lei, Z. Zhang, B. Pan, C. Ling, and L. Zhao, “Grag: Graph retrieval-augmented generation,” arXiv preprint arXiv:2405.16506, 2024.
J. Delile, S. Mukherjee, A. Van Pamel, and L. Zhukov, “Graph-based retriever captures the long tail of biomedical knowledge,” arXiv preprint arXiv:2402.12352, 2024.
C. Mavromatis and G. Karypis, “Gnn-rag: Graph neural retrieval for large language model reasoning,” arXiv preprint arXiv:2405.20139, 2024.
J. Jiang, K. Zhou, Z. Dong, K. Ye, W. X. Zhao, and J.-R. Wen, “Structgpt: A general framework for large language model to reason over structured data,” arXiv preprint arXiv:2305.09645, 2023.
J. Dong, Q. Zhang, X. Huang, K. Duan, Q. Tan, and Z. Jiang, “Hierarchy-aware multi-hop question answering over knowledge graphs,” in Proceedings of the ACM Web Conference 2023, ser. WWW ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 2519–2527. [Online]. DOI: 10.1145/3543507.3583376
M. Zhang, M. Sun, P. Wang, S. Fan, Y. Mo, X. Xu, H. Liu, C. Yang, and C. Shi, “Graphtranslator: Aligning graph model to large language model for open-ended tasks,” Proceedings of the ACM on Web Conference 2024, 2024. [Online]. Available: [link]
J. Martinez-Gil, “A survey on legal question–answering systems,” Computer Science Review, 2021. [Online]. Available: [link]
F. Dai, Z. Zhao, C. Sun, and B. Li, “Intelligent audit question answering system based on knowledge graph and semantic similarity,” 2022 11th International Conference of Information and Communication Technology (ICTech)), pp. 125–132, 2022. [Online]. Available: [link]
F. Sovrano, M. Palmirani, and F. Vitali, “Legal knowledge extraction for knowledge graph based question-answering,” in International Conference on Legal Knowledge and Information Systems, 2020. [Online]. Available: [link]
W. Huang, J. Jiang, Q. Qu, and M. Yang, “Aila: A question answering system in the legal domain,” in International Joint Conference on Artificial Intelligence, 2020. [Online]. Available: [link]
E. Filtz, S. Kirrane, and A. Polleres, “The linked legal data landscape: linking legal data across different countries,” Artificial Intelligence and Law, vol. 29, pp. 485 – 539, 2021. [Online]. Available: [link]
A. Thomas and S. Sangeetha, “Knowledge graph based question-answering system for effective case law analysis,” in International Conference on Frontiers in Intelligent Computing: Theory and Applications, 2021. [Online]. Available: [link]
S. Es, J. James, L. Espinosa Anke, and S. Schockaert, “RAGAS: Automated evaluation of retrieval augmented generation,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, N. Aletras and O. De Clercq, Eds. St. Julians, Malta: Association for Computational Linguistics, Mar. 2024, pp. 150–158. [Online]. Available: [link]
D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,” arXiv preprint arXiv:2104.10350, 2021.
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024.
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrievalaugmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020.
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024.
X. He, Y. Tian, Y. Sun, N. V. Chawla, T. Laurent, Y. LeCun, X. Bresson, and B. Hooi, “G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,” arXiv preprint arXiv:2402.07630, 2024.
W. Fan, Y. Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6491–6501.
Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, 2023.
Y. Huang and J. Huang, “A survey on retrieval-augmented text generation for large language models,” arXiv preprint arXiv:2404.10981, 2024.
S. Wu, Y. Xiong, Y. Cui, H. Wu, C. Chen, Y. Yuan, L. Huang, X. Liu, T.-W. Kuo, N. Guan et al., “Retrieval-augmented generation for natural language processing: A survey,” arXiv preprint arXiv:2407.13193, 2024.
P. Zhao, H. Zhang, Q. Yu, Z. Wang, Y. Geng, F. Fu, L. Yang, W. Zhang, and B. Cui, “Retrieval-augmented generation for ai-generated content: A survey,” arXiv preprint arXiv:2402.19473, 2024.
H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of retrieval-augmented generation: A survey,” arXiv preprint arXiv:2405.07437, 2024.
M. Yasunaga, H. Ren, A. Bosselut, P. Liang, and J. Leskovec, “Qagnn: Reasoning with language models and knowledge graphs for question answering,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 535–546.
D. Taunk, L. Khanna, S. V. P. K. Kandru, V. Varma, C. Sharma, and M. Tapaswi, “Grapeqa: Graph augmentation and pruning to enhance question-answering,” in Companion Proceedings of the ACM Web Conference 2023, 2023, pp. 1138–1144.
S. Li, Y. Gao, H. Jiang, Q. Yin, Z. Li, X. Yan, C. Zhang, and B. Yin, “Graph reasoning for question answering with triplet retrieval,” arXiv preprint arXiv:2305.18742, 2023.
Y. Huang, Y. Li, Y. Xu, L. Zhang, R. Gan, J. Zhang, and L. Wang, “Mvp-tuning: Multi-view knowledge retrieval with prompt tuning for commonsense reasoning,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 13 417–13 432.
Y. Hu, Z. Lei, Z. Zhang, B. Pan, C. Ling, and L. Zhao, “Grag: Graph retrieval-augmented generation,” arXiv preprint arXiv:2405.16506, 2024.
J. Delile, S. Mukherjee, A. Van Pamel, and L. Zhukov, “Graph-based retriever captures the long tail of biomedical knowledge,” arXiv preprint arXiv:2402.12352, 2024.
C. Mavromatis and G. Karypis, “Gnn-rag: Graph neural retrieval for large language model reasoning,” arXiv preprint arXiv:2405.20139, 2024.
J. Jiang, K. Zhou, Z. Dong, K. Ye, W. X. Zhao, and J.-R. Wen, “Structgpt: A general framework for large language model to reason over structured data,” arXiv preprint arXiv:2305.09645, 2023.
J. Dong, Q. Zhang, X. Huang, K. Duan, Q. Tan, and Z. Jiang, “Hierarchy-aware multi-hop question answering over knowledge graphs,” in Proceedings of the ACM Web Conference 2023, ser. WWW ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 2519–2527. [Online]. DOI: 10.1145/3543507.3583376
M. Zhang, M. Sun, P. Wang, S. Fan, Y. Mo, X. Xu, H. Liu, C. Yang, and C. Shi, “Graphtranslator: Aligning graph model to large language model for open-ended tasks,” Proceedings of the ACM on Web Conference 2024, 2024. [Online]. Available: [link]
J. Martinez-Gil, “A survey on legal question–answering systems,” Computer Science Review, 2021. [Online]. Available: [link]
F. Dai, Z. Zhao, C. Sun, and B. Li, “Intelligent audit question answering system based on knowledge graph and semantic similarity,” 2022 11th International Conference of Information and Communication Technology (ICTech)), pp. 125–132, 2022. [Online]. Available: [link]
F. Sovrano, M. Palmirani, and F. Vitali, “Legal knowledge extraction for knowledge graph based question-answering,” in International Conference on Legal Knowledge and Information Systems, 2020. [Online]. Available: [link]
W. Huang, J. Jiang, Q. Qu, and M. Yang, “Aila: A question answering system in the legal domain,” in International Joint Conference on Artificial Intelligence, 2020. [Online]. Available: [link]
E. Filtz, S. Kirrane, and A. Polleres, “The linked legal data landscape: linking legal data across different countries,” Artificial Intelligence and Law, vol. 29, pp. 485 – 539, 2021. [Online]. Available: [link]
A. Thomas and S. Sangeetha, “Knowledge graph based question-answering system for effective case law analysis,” in International Conference on Frontiers in Intelligent Computing: Theory and Applications, 2021. [Online]. Available: [link]
S. Es, J. James, L. Espinosa Anke, and S. Schockaert, “RAGAS: Automated evaluation of retrieval augmented generation,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, N. Aletras and O. De Clercq, Eds. St. Julians, Malta: Association for Computational Linguistics, Mar. 2024, pp. 150–158. [Online]. Available: [link]
D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,” arXiv preprint arXiv:2104.10350, 2021.
Publicado
22/10/2025
Como Citar
OLIVEIRA, Vinícius Teles; LIMA, Maurício Rodrigues; TELES, Sávio; DIAS, Elisângela Silva.
J-KGRAG: A Hybrid Retrieval-Augmented Generation Architecture for Legal Norm Understanding with Knowledge Graphs. In: CONGRESSO LATINO-AMERICANO DE SOFTWARE LIVRE E TECNOLOGIAS ABERTAS (LATINOWARE), 22. , 2025, Foz do Iguaçu/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 202-209.
DOI: https://doi.org/10.5753/latinoware.2025.16269.
