Challenges of Knowledge Graphs: a proposal for evaluating OpenIE systems

• Samuel Rios da Silva UFBA
• Aline Athaydes UFBA
• Babacar Mane UFBA
• Daniela Barreiro Claro UFBA
• Marlo Souza UFBA
• Fernando H. de A. Moraes Neto UFBA
• Larrissa Dantas UFBA
• Rerisson Cavalcante UFBA

DOI: https://doi.org/10.5753/stil.2025.37878

Abstract

Open Information Extraction (OpenIE) faces challenges in evaluating its models. With the use of traditional metrics from other areas and the absence of a gold standard corpus, the difficulty arises in evaluating all possible extractions generated by the models. In this work, we propose a comparative evaluation method for different OpenIE models focused on the Portuguese language using knowledge graphs. The results obtained show that models capable of generating a greater number of accurate triples tend to deliver better performance, highlighting the importance of balancing quantity and quality in the OpenIE task.

References

Cabral, B., Souza, M., and Claro, D. B. (2022). Portnoie: A neural framework for open information extraction for the portuguese language. In International Conference on Computational Processing of the Portuguese Language, pages 243–255. Springer.

Claro, D. B., Santos, J., Souza, M., Vieira, R., and Pinheiro, V. (2024). Extração de informação. In Caseli, H. M. and Nunes, M. G. V., editors, Processamento de Linguagem Natural: Conceitos, Técnicas e Aplicações em Português, book chapter 20. BPLN, 2 edition.

Glauber, R. and Barreiro Claro, D. (2018). A systematic mapping study on open information extraction. Expert Systems with Applications, 112:372–387.

Oliveira, L., Claro, D. B., and Souza, M. (2023). Dptoie: a portuguese open information extraction based on dependency analysis. Artificial Intelligence Review, 56(7):7015–7046.

Paulheim, H. (2017). Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web, 8(3):489–508.

Queiroz, B., Cavalcante, R., and Claro, D. (2023). Desafios da tarefa de extração de informação aberta: uma abordagem metodológica de um corpus automatizado até o corpus manual. In Anais do XIV Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 388–392, Porto Alegre, RS, Brasil. SBC.