LLM-Text2KG: Um Pipeline Orientado por Ontologias para Construção e Avaliação de Grafos de Conhecimento Extraídos de Documentos Institucionais

• Giovanna Verissimo UFC
• Julio Cesar Marinho UFC
• Marlon Duarte UFC
• Amanda D. Pires Venceslau UFC
• Vania Vidal UFC
• Wellington Franco UFC
• Carlos Caminha UFC

DOI: https://doi.org/10.5753/webmedia.2026.30738

Resumo

Institutional documents, such as meeting minutes, contain valuable organizational knowledge, but their unstructured nature hinders systematic analysis. This work presents LLM-Text2KG, an ontology-guided pipeline for constructing knowledge graphs from institutional documents. The pipeline was evaluated on ten meeting minutes from the Acaraú River Basin Committee using six large language models under zero-shot and few-shot strategies. The evaluation combined Precision, Recall, and F1-score against a manually annotated gold standard, ontology-conformity validation, competency questions expressed as SPARQL queries, and an ablation of the ontology representation. Gemma 4 31B in few-shot achieved the best performance, with F1-score of 0.650 for entities and 0.498 for relations, and 99.37% ontology conformity. The competency questions achieved 41.0% overall coverage. Replacing the Turtle ontology with a structured textual list reduced relation F1-score from 0.464 to 0.102 in zero-shot and from 0.498 to 0.265 in few-shot. The results indicate that ontology representation influences extraction quality and that semantic relation extraction remains the main challenge.

Palavras-chave: Knowledge Graph, Large Language Models, Prompt Engineering, Ontology-Guided Extraction, Competency Questions

Referências

Camila Bezerra, Fred Freitas, and Filipe Santana. 2013. Evaluating Ontologies with Competency Questions. In 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), Vol. 3. 284–285. DOI: 10.1109/WI-IAT.2013.199

Ricardo Campos, José Pedro Evans, José Miguel Isidro, Miguel Marques, Luís Filipe Cunha, Alípio Jorge, Sérgio Nunes, and Nuno Guimarães. 2026. NLP for Local Governance Meeting Records: A Focus Article on Tasks, Datasets, Metrics and Benchmark. arXiv:2602.08162 [cs.CL] [link]

Keyin Chen, Rui Yao, Zhichuan Tang, Anqi Yao, and Xiaoneng Jin. 2025. Bridging cultural gaps in product design: A crosscultural supervisor approach using large language model. Design and Artificial Intelligence 1, 1 (2025), 100004. DOI: 10.1016/j.daai.2025.100004

Eric Mateus Dias, Zoraide Pessoa, and Rylanneive Teixeira. 2023. ADAPTIVE GOVERNANCE AND WATER SECURITY IN THE CONTEXT OF CLIMATE CHANGE IN THE SEMI-ARID. Mercator 21 (2023). DOI: 10.4215/rm2022.e21025

Bradley Efron and Robert J. Tibshirani. 1994. An Introduction to the Bootstrap (1 ed.). Chapman and Hall/CRC. DOI: 10.1201/9780429246593

Michael Grüninger and Mark S. Fox. 1995. The Role of Competency Questions in Enterprise Engineering. Springer US, Boston, MA, 22–31. DOI: 10.1007/978-0-387-34847-6_3

Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. (2020). DOI: 10.5281/zenodo.1212303

Haoyu Huang, Chong Chen, Zeang Sheng, Yang Li, and Wentao Zhang. 2025. Can LLMs be Good Graph Judge for Knowledge Graph Construction?. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 10929–10948. DOI: 10.18653/v1/2025.emnlp-main.554

Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. 2022. A Survey on Knowledge Graphs: Representation, Acquisition, and Applications. IEEE Transactions on Neural Networks and Learning Systems 33, 2 (2022), 494–514. DOI: 10.1109/TNNLS.2021.3070843

Samira Khorshidi, Azadeh Nikfarjam, Suprita Shankar, Yisi Sang, Yash Govind, Hyun Jang, Ali Kasgari, Alexis McClimans, Mohamed Soliman, Vishnu Konda, Ahmed Fakhry, and Xiaoguang Qi. 2025. ODKE+: Ontology-Guided Open-Domain Knowledge Extraction with LLMs. arXiv:2509.04696 [cs.CL] [link]

Jonathan Larson and Steven Truitt. 2024. GraphRAG: Unlocking LLM Discovery on Narrative Private Data. Microsoft Research Blog. [link] Acesso em: jun. 2026.

Yancheng Ling, Zhenlin Qin, and Zhenliang Ma. 2026. A review of knowledge graph construction using large language models in transportation: Problems, methods, and challenges. Transportation Research Part C: Emerging Technologies 183, Article 105428 (Feb. 2026), 105428 pages. DOI: 10.1016/j.trc.2025.105428

Bingchen Liu, Yuanyuan Fang, Naixing Xu, Shihao Hou, Xin Li, and Qian Li. 2025. Large Language Models for Knowledge Graph Embedding: A Survey. Mathematics 13, 14 (2025). DOI: 10.3390/math13142244

Nandana Mihindukulasooriya, Sanju Tiwari, Carlos F. Enguix, and Kusum Lata. 2023. Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text. In The Semantic Web – ISWC 2023, Terry R. Payne, Valentina Presutti, Guilin Qi, María Poveda-Villalón, Giorgos Stoilos, Laura Hollink, Zoi Kaoudi, Gong Cheng, and Juanzi Li (Eds.). Springer Nature Switzerland, Cham, 247–265.

Belinda Mo, Kyssen Yu, Joshua Kazdan, Proud Mpala, Lisa Yu, Charilaos Kanatsoulis, and Sanmi Koyejo. 2025. KGGen: Extracting Knowledge Graphs from Plain Text with Language Models. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38. Curran Associates, Inc., 30092–30115. [link]

Itai Mondshine, Tzuf Paz-Argaman, and Reut Tsarfaty. 2025. Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs. In Findings of the Association for Computational Linguistics: NAACL 2025, Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 1331–1354. DOI: 10.18653/v1/2025.findings-naacl.73

Glaice Kelly Q. Monfardini, Jordana S. Salamon, and Monalessa P. Barcellos. 2023. Use ofnbsp;Competency Questions innbsp;Ontology Engineering: A Survey. In Conceptual Modeling: 42nd International Conference, ER 2023, Lisbon, Portugal, November 6–9, 2023, Proceedings (Lisbon, Portugal). Springer-Verlag, Berlin, Heidelberg, 45–64. DOI: 10.1007/978-3-031-47262-6_3

Daniel Sanford Moreira. 2014. O Comitê de bacia como instrumento de democratização dos recursos hídricos. Um estudo de caso: a atuação do Comitê da Bacia do Acaraú – Ceará. , 91 pages. [link]

Natalya F. Noy and Deborah L. McGuinness. 2001. Ontology Development 101: A Guide to Creating Your First Ontology. Technical Report KSL-01-05 and SMI-2001-0880. Stanford Knowledge Systems Laboratory and Stanford Medical Informatics, Stanford, CA. [link]

Vayianos Pertsas and Panos Constantopoulos. 2023. Ontology-Driven Extraction of Contextualized Information from Research Publications.. In KEOD. 108–118.

Julien Plu, Oscar Moreno Escobar, Edouard Trouillez, Axelle Gapin, Pasquale Lisena, Thibault Ehrhart, and Raphaël Troncy. 2025. Text2KGBench-LettrIA: A refined benchmark for Text2Graph systems. In KBC-LM and LM-KBC Challenge (Knowledge Base Construction from Pre-trained Language Models) at ISWC 2025, 24th International Semantic Web Conference, 2-6 November 2025, Nara, Japan, CEUR (Ed.). Nara. [link] CEUR.

André Regino and Julio Reis. 2025. Enhancing Knowledge Graphs with Large Language Models: Contributions to E-commerce Question Answering Systems. In Anais Estendidos do XXXI Simpósio Brasileiro de Sistemas Multimídia e Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 21–22. DOI: 10.5753/webmedia_estendido.2025.16346

Ridho Reinanda, Edgar Meij, and Maarten de Rijke. 2020. Knowledge Graphs: An Information Retrieval Perspective. Found. Trends Inf. Retr. 14, 4 (Oct. 2020), 289–444. DOI: 10.1561/1500000063

Marlene Yara Tenório Soares and Rodrigo Guimarães de Carvalho. 2022. Gestão dos Recursos Hídricos e Conflitos no Semiárido: Estudo do Caso do Açude Lucrécia, Estado do Rio Grande do Norte. Revista GeoInterações 6, 1 (Dec. 2022), 57–76. [link]

Milena Trajanoska, Riste Stojanov, and Dimitar Trajanov. 2023. Enhancing Knowledge Graph Construction Using Large Language Models. arXiv:2305.04676 [cs.CL] [link]