Agentic Benchmark Generation and Factorial Evaluation of Text-to-SQL Architectures on an Industrial Database

• Thiago Pedroso de Jesus UFG / CEIA
• Victor Arantes UFG / CEIA
• Fernanda Färber UFG / CEIA
• Artur Matos UFG / CEIA
• Vinicius Aguiar UFG / CEIA
• Sávio Teles UFG / CEIA
• Anderson da Silva Soares UFG / CEIA

DOI: https://doi.org/10.5753/eniac.2026.31523

Resumo

Deploying Text-to-SQL in specialized industrial domains demands an evaluation benchmark grounded in real operational needs and an understanding of which architectural choices truly improve accuracy. Human-annotated benchmarks are prohibitively expensive, and current synthetic generators read only the schema, never the data, missing the quirks that make industrial questions hard. We present two contributions. First, an agentic benchmark generation pipeline in which a ReAct agent with five database-exploration tools formulates each question/SQL pair from observed data, enforcing quality through execution validation, vocabulary gates, and dual-layer deduplication; it expands 37 expert-validated seeds into a 169-pair Portuguese benchmark without paid annotation. Second, a controlled factorial study that fixes the model, benchmark, and metadata to isolate how schema presentation and tool access affect six architectures. We find that full-schema access and tool use add accuracy independently, that semantic schema linking helps simple but starves medium-complexity queries, and that self-correction and multi-agent decomposition do not pay off; above all, 57% of questions fail across every architecture, showing that the binding constraint is domain knowledge, not architectural sophistication.

Palavras-chave: Text-to-SQL, Agentic benchmark generation, Schema linking, Factorial evaluation, Industrial NLP, LLM agents

Referências

Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Morcos, A. S. (2023). SemDeDup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540.

Caferoğlu, H. A., Çelik, M. S., and Ulusoy, O. (2025). SING-SQL: A synthetic data generation framework for in-domain text-to-SQL translation. arXiv preprint arXiv:2509.25672.

Cao, B., Liao, W., Sun, Y., Fang, D., Li, H., and Lam, W. (2026). APEX-SQL: Talking to the data via agentic exploration for text-to-SQL. arXiv preprint arXiv:2602.16720.

Cao, Z., Zheng, Y., Fan, Z., Zhang, X., Chen, W., and Bai, X. (2024). RSL-SQL: Robust schema linking in text-to-SQL generation. arXiv preprint arXiv:2411.00073.

Cheng, S., Cheng, Q., Jin, L., Yi, L., and Zhang, G. (2025). SQLord: A robust enterprise text-to-SQL solution via reverse data generation and workflow decomposition. In Companion Proceedings of WWW 2025, pages 919–923. Association for Computing Machinery.

Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2024). Text-to-SQL empowered by large language models: A benchmark evaluation. Proceedings of the VLDB Endowment, 17(5):1132–1145.

Granado, F., Lotufo, R., and Pereira, J. (2025). RAISE: Reasoning agent for interactive SQL exploration. arXiv preprint arXiv:2506.01273.

Huang, Z. and Wu, E. (2024). Cocoon: Semantic table profiling using large language models. In Proceedings of the 2024 Workshop on Human-In-the-Loop Data Analytics (HILDA), pages 1–7. Association for Computing Machinery.

José, M. A. and Cozman, F. G. (2021). mRAT-SQL+GAP: A portuguese text-to-SQL transformer. In Proceedings of the Brazilian Conference on Intelligent Systems (BRACIS), volume 13074 of LNAI, pages 511–525. Springer.

Kim, H., Jeon, T., Choi, S., Choi, S., and Cho, H. (2025). FLEX: Expert-level false-less execution metric for text-to-SQL benchmark. In Proceedings of NAACL 2025, pages 4448–4475. Association for Computational Linguistics.

Li, H., Wu, S., Zhang, X., Huang, X., Zhang, J., Jiang, F., Wang, S., Zhang, T., Chen, J., Shi, R., Chen, H., and Li, C. (2025). OmniSQL: Synthesizing high-quality text-to-SQL data at scale. arXiv preprint arXiv:2503.02240.

Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., Huo, N., Zhou, X., Chenhao, M., Li, G., Chang, K., Huang, F., Cheng, R., and Li, Y. (2023). Can LLM already serve as a database interface? A BIg bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 42330–42357.

Mitra, A., Del Corro, L., Zheng, G., Mahajan, S., Rouhana, D., Codas, A., Lu, Y., Chen, W.-g., Vrousgos, O., Rosset, C., Silva, F., Khanpour, H., Lara, Y., and Awadallah, A. (2024). AgentInstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502.

Pan, X., Tao, C., Bai, J., Gao, J., Tao, Z., Zhou, X., Cheung, G., and Ma, S. (2026). EvolSQL: Structure-aware evolution for scalable text-to-SQL data synthesis. arXiv preprint arXiv:2601.04875.

Pourreza, M. and Rafiei, D. (2023). DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 36339–36348.

Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 8634–8652.

Shkapenyuk, V., Srivastava, D., Johnson, T., and Ghane, P. (2025). Automatic metadata extraction for text-to-SQL. arXiv preprint arXiv:2505.19988.

Talaei, S., Pourreza, M., Chang, Y.-C., Mirhoseini, A., and Saberi, A. (2024). CHESS: Contextual harnessing for efficient SQL synthesis. arXiv preprint arXiv:2405.16755.

Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., Sun, X., and Li, Z. (2025). MAC-SQL: A multi-agent collaborative framework for text-to-SQL. In Proceedings of COLING 2025, pages 540–557. Association for Computational Linguistics.

Wang, Z., Zheng, Y., Cao, Z., Zhang, X., Wei, Z., Fu, P., Luo, Z., Chen, W., and Bai, X. (2026). AutoLink: Autonomous schema exploration and expansion for scalable schema linking in text-to-SQL at scale. In Proceedings of AAAI 2026. AAAI Press.

Xie, W., Dai, Y., and Jiang, W. (2025). SDE-SQL: Enhancing text-to-SQL generation in large language models via self-driven exploration with SQL probes. arXiv preprint arXiv:2506.07245.

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR).

Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3911–3921. Association for Computational Linguistics.