Linguistic Analysis of Neural-to-Symbolic Translation Failures in Knowledge Graph-Based Fact Checking
Resumo
Large Language Models are increasingly paired with Knowledge Graphs to verify factual claims, yet such systems are typically evaluated at the response level, leaving open which linguistic phenomena break the neural-to-symbolic translation from a sentence to a verifiable triple. We study this step in isolation with a within-fact design: 26 Wikidata-grounded facts, each expressed across eight linguistic conditions. gpt-4o-mini is evaluated across the full design, and Claude Haiku 4.5 provides a confirmatory comparison on four diagnostic conditions. A symbolic layer then verifies each extracted triple by SPARQL ASK; keeping the fact constant attributes every failure to the linguistic surface form. For gpt-4o-mini, failures concentrate in constructions that structurally displace the target argument from its canonical position, including exceptor phrases, where it extracts the head noun rather than the excluded entity (46.2% accuracy). Claude Haiku 4.5 likewise performs poorly on exceptor phrases (42.3%), but its errors take a different form: it declines to extract a triple in 15 of 26 cases rather than substituting the head noun. Thus, the exceptor construction is difficult across the two models, whereas the head-noun-substitution signature is model-specific. Neither model substitutes a negated entity’s true-valued counterpart in the tested negation conditions. These results are descriptive and do not establish general performance across models or operational fact-checking settings.
Referências
Cao, N. D., Izacard, G., Riedel, S., and Petroni, F. (2021). Autoregressive entity retrieval.
FitzGerald, J. et al. (2024). MASSIVE multilingual abstract meaning representation: A dataset and baselines for hallucination detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.
Herron, S. et al. (2025). On the potential of logic and reasoning in neurosymbolic systems using OWL-based knowledge graphs. SAGE Open Computer Science.
Jurafsky, D. and Martin, J. H. (2024). Speech and Language Processing. 3rd (draft) edition. Available at [link].
Lavrinovics, E., Biswas, R., Bjerva, J., and Hose, K. (2025). Knowledge graphs, large language models, and hallucinations: An nlp perspective. Journal of Web Semantics, 85:100844.
Opitz, J., Wein, S., and Schneider, N. (2025). Natural language processing relies on linguistics. Computational Linguistics, 51(3):1009–1032.
Petcu, R., Bhargav, S., de Rijke, M., and Kanoulas, E. (2025). A comprehensive taxonomy of negation for NLP and neural retrievers. [link], pages 15511–15533.
Rani, M., Mishra, B. K., and Thakker, D. (2025). Advancing symbolic integration in large language models: Beyond conventional neurosymbolic ai.
Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912. Association for Computational Linguistics.
Sansford, H. et al. (2024). GraphEval: A knowledge-graph based LLM hallucination evaluation framework. arXiv preprint arXiv:2407.10793.
Vrandečić, D. and Krötzsch, M. (2014). Wikidata: a free collaborative knowledgebase. Commun. ACM, 57(10):78–85.
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., and Bowman, S. R. (2020). BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392.
Wu, L., Petroni, F., Josifoski, M., Riedel, S., and Zettlemoyer, L. (2020). Scalable zero-shot entity linking with dense entity retrieval. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397–6407, Online. Association for Computational Linguistics.
