SAGE-SQL: An Agentic Generator for Synthetic Text-to-SQL Benchmarks Grounded in Real Industrial Schemas
Resumo
Industrial Text-to-SQL deployments target schemas that academic benchmarks like Spider and BIRD do not exercise: foreign keys encoded only as naming conventions, sparsely populated columns, and domain-specific operational vocabulary. Synthetic benchmark generators have matured rapidly, yet every published generator we surveyed shares one structural feature: synthesis proceeds in fixed sequential stages, with execution used at most as a post-hoc filter. We introduce SAGE-SQL, an agentic generator in which a fresh ReAct-style agent with a five-tool toolkit explores a real production database during synthesis, constrained by a deterministic 15-template by 4-style schedule and inline execution-grounded gates. As a first, deliberately focused case study, we compare it against controlled re-implementations of OmniSQL and SQaLe on a single industrial schema (a regulated electric energy distribution utility) under a four-dimensional pointwise LLM-as-judge and a five-metric textual diversity panel. SAGE-SQL attains the highest mean schema grounding (4.80/5) and ranks first on four of five textual diversity metrics, while SQaLe attains the highest question quality and OmniSQL the lowest per-pair cost. Our contributions are the agentic generator paradigm, an evaluation framework that exposes textual mode collapse invisible to SQL duplicate metrics, and a single-schema characterization of when each paradigm is preferable, framed as a preliminary, methodology-focused study.
Referências
Deng, M., Ramachandran, A., Xu, C., Hu, L., Yao, Z., Datta, A., and Zhang, H. (2025). ReFoRCE: A text-to-SQL agent with self-refinement, consensus enforcement, and column exploration. arXiv preprint arXiv:2502.00675.
Kim, H., Jeon, T., Choi, S., Choi, S., and Cho, H. (2025). FLEX: Expert-level false-less EXecution metric for text-to-SQL benchmark. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 4448–4475.
Li, H., Wu, S., Zhang, X., Huang, X., Zhang, J., Jiang, F., Wang, S., Zhang, T., Chen, J., Shi, R., Chen, H., and Li, C. (2025). OmniSQL: Synthesizing high-quality text-to-SQL data at scale. Proceedings of the VLDB Endowment, 18(11):4695–4709.
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2016). A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Cao, R., Geng, R., Huo, N., Zhou, X., Ma, C., Li, G., Chang, K. C. C., Huang, F., Cheng, R., and Li, Y. (2023). Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems (NeurIPS).
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Mitra, A., Corro, L. D., Zheng, G., Mahajan, S., Rouhana, D., Codas, A., Lu, Y., ge Chen, W., Vrousgos, O., Rosset, C., Silva, F., Khanpour, H., Lara, Y., and Awadallah, A. (2024). AgentInstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502.
Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024).
Pei, W., Hou, S., Li, B., Chen, H., Shi, Z., and Luo, Y. (2026). ROSE: An intent-centered evaluation metric for NL2SQL. Accepted to ACL 2026 (Main Conference).
Wolff, C., Gomm, D., and Hulsebos, M. (2025). SQaLe: A large text-to-SQL corpus grounded in real schemas. arXiv preprint arXiv:2602.22223.
Yamauchi, Y., Yano, T., and Oyamada, M. (2025). An empirical study of LLM-as-a-Judge: How design choices impact evaluation reliability. arXiv preprint arXiv:2506.13639.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR).
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. (2018). Texygen: A benchmarking platform for text generation models. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.
