Neuro-Symbolic Graph-RAG for Academic Advising: A Three-Cycle Evaluation of a University Web Chatbot

• Leonardo Arazo de Oliveira Araújo UNIFESP
• Didier A. Vega-Oliveros UNIFESP

DOI: https://doi.org/10.5753/webmedia_estendido.2026.34475

Resumo

Retrieval-Augmented Generation (RAG) and its graph-based extensions improve the grounding of Large Language Models, but do not guarantee logical consistency when answers must satisfy domain constraints composed of several facts. This work presents a bidirectional neuro-symbolic validation framework for Graph-RAG, deployed as an academic advising chatbot at a Brazilian public university. It couples First-Order Logic reasoning over a domain knowledge graph with an Annotated Datalog engine, operating in three modes: pre-generation symbolic enrichment, post-hoc validation with corrective rewriting, and direct symbolic execution. We evaluated the system in three iterative cycles covering accessibility conformance, usability testing with ten participants, and answer-consistency benchmarking, where it reaches 84.2% strict and 87.7% weighted accuracy, an 87% relative error reduction over a baseline using the language model alone and a gain of 27.2 percentage points over Graph-RAG without symbolic validation. Moreover, we replicated the study on a second institution’s curriculum, with no changes to the engine or the rules, and it showed similar results and the same ablation ordering.

Palavras-chave: Neuro-Symbolic AI, Graph-RAG, Knowledge Graphs, Large Language Models, Web Accessibility, Usability Evaluation

Referências

Dyuman Aditya, Kaustuv Mukherji, Srikar Balasubramanian, Abhiraj Chaudhary, and Paulo Shakarian. 2023. PyReason: Software for Open World Temporal Logic. In AAAI Spring Symposium.

Prerna Agarwal and Srikanta Bedathur. 2025. A Zero-Shot Neuro-Symbolic Approach for Complex Knowledge Graph Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2025. 11514–11527. DOI: 10.18653/v1/2025.findings-emnlp.617

Akari Asai, ZeqiuWu, YizhongWang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations (ICLR).

John Brooke. 1996. SUS: A Quick and Dirty Usability Scale. In Usability Evaluation in Industry, Patrick W. Jordan, Bruce Thomas, Bernard A. Weerdmeester, and Ian L. McClelland (Eds.). Taylor and Francis, 189–194.

Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, Daoyu Wang, and Enhong Chen. 2025. A Survey on Knowledge-Oriented Retrieval-Augmented Generation. arXiv preprint arXiv:2503.10677 (2025).

Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37–46. DOI: 10.1177/001316446002000104

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. Chain-of-Verification Reduces Hallucination in Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024.

Artur S. d’Avila Garcez and Luis C. Lamb. 2023. Neurosymbolic AI: The 3rd Wave. Artificial Intelligence Review 56 (2023), 12387–12406. DOI: 10.1007/s10462-023-1048-w

Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. 2024. Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based Retrofitting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38.

Haoyu Han, Harry Shomer, YuWang, Yongjia Lei, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, and Jiliang Tang. 2025. RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. arXiv preprint arXiv:2502.11371 (2025).

Jasper Linders and Jakub M. Tomczak. 2025. Knowledge Graph-Extended Retrieval Augmented Generation for Question Answering. Applied Intelligence 55 (2025), 1102. DOI: 10.1007/s10489-025-06885-5

Liangming Pan, Alon Albalak, XinyiWang, and William YangWang. 2023. Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023.

Lorena Amanda Quincoso Lugones, Christopher Kverne, Nityam Sharadkumar Bhimani, Ana Carolina Oliveira, Agoritsa Polyzou, Christine Lisetti, and Janki Bhimani. 2026. Aurora: Neuro-Symbolic AI Driven Advising Agent. In Proceedings of the 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26). Association for Computing Machinery. DOI: 10.1145/3748522.3779850

Junde Wu, Jiayuan Zhu, Yunli Qi, Jingwei Chen, Min Xu, Filippo Menolascina, and Vicente Grau. 2024. Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation. arXiv preprint arXiv:2408.04187 (2024).

Xiangrong Zhu, Yuexiang Xie, Yi Liu, Yaliang Li, and Wei Hu. 2025. Knowledge Graph-Guided Retrieval Augmented Generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. 8912–8924. DOI: 10.18653/v1/2025.naacl-long.449