Text-to-SQL Beyond Benchmarks: A Case Study on Large-Scale Deployment

• Lucas Nobre UFG
• Vinicius Aguiar UFG
• José Oliveira UFG
• Lucas Alves UFG
• Anderson Soares UFG
• Sávio Teles UFG

DOI: https://doi.org/10.5753/eniac.2026.31596

Resumo

Most Text-to-SQL evaluation evidence comes from academic benchmarks with small schemas and curated metadata, which differ from the databases these systems face in deployment. This study evaluates an agentic Text-to-SQL system deployed over open data from ANEEL, the Brazilian electricity regulator, a 140-table Brazilian-Portuguese database whose schema metadata is extracted automatically from dataset documentation. The agent is compared against a non-agentic one-shot baseline, with and without exploration tools, across schema scopes of 10, 50, and 140 tables, with BIRD-PT as an academic reference point. Each answer is scored under strict execution accuracy, relaxed execution accuracy, and a semantic-match judge. The agent reaches 63.56% strict execution accuracy on BIRD-PT and 15.01% on ANEEL under the same model, a gap of at least 36 percentage points on every metric. The correct table is retrieved for 60% of ANEEL questions against nearly 100% BIRD questions, and accuracy falls further during query construction. Across all schema scopes, strict execution accuracy ranks the baseline above the agent while relaxed execution accuracy and semantic match rank the agent above it, so the metric chosen decides which system a practitioner would select.

Referências

Cen, J., Liu, J., Li, Z., and Wang, J. (2025). Sqlfixagent: Towards semantic-accurate textto-sql parsing via consistency-enhanced multi-agent collaboration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 49–57.

de Carvalho, L. F. C., Júnior, P. S. d. S., and de Oliveira, H. T. A. (2025). Benchmarking Large Language Models for Text-to-SQL in Brazilian Portuguese and English. In Simpósio Brasileiro de Tecnologia Da Informação e Da Linguagem Humana (STIL), pages 101–112. SBC.

Floratou, A., Psallidas, F., Zhao, F., Deep, S., Hagleither, G., Tan, W., Cahoon, J., Alotaibi, R., Henkel, J., Singla, A., Grootel, A. V., Chow, B., Deng, K., Lin, K., Campos, M., Emani, K. V., Pandit, V., Shnayder, V., Wang, W., and Curino, C. (2024). Nl2sql is a solved problem... not! In Conference on Innovative Data Systems Research.

Li, H., Zhang, J., Li, C., and Chen, H. (2023). Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13067–13075.

Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., Huo, N., et al. (2024). Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36.

Nobre, L., Sousa, S., Teles, S., and Soares, A. (2026). Language effects in text-to-SQL across English and Portuguese. In Souza, M., de Dios-Flores, I., Santos, D., Freitas, L., Souza, J. W. d. C., and Ribeiro, E., editors, Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1, pages 270–280, Salvador, Brazil. Association for Computational Linguistics.

Pham, K. T., Nguyen, T. H., Jo, J., Nguyen, Q. V. H., and Nguyen, T. T. (2025). Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents.

Pourreza, M., Li, H., Sun, R., Chung, Y., Talaei, S., Kakkar, G. T., Gan, Y., Saberi, A., Ozcan, F., and Arik, S. O. (2024). CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL.

Pourreza, M. and Rafiei, D. (2023). DIN-SQL: Decomposed In-context learning of text-to-SQL with self-correction. Advances in Neural Information Processing Systems, 36:36339–36348.

Shkapenyuk, V., Srivastava, D., Johnson, T., and Ghane, P. (2025). Automatic Metadata Extraction for Text-to-SQL.

Song, Y., Ezzini, S., Tang, X., Lothritz, C., Klein, J., Bissyandé, T., Boytsov, A., Ble, U., and Goujon, A. (2024). Enhancing text-to-sql translation for financial system design. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, pages 252–262.

Talaei, S., Pourreza, M., Chang, Y.-C., Mirhoseini, A., and Saberi, A. (2024). CHESS: Contextual Harnessing for Efficient SQL Synthesis.

Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., Sun, X., et al. (2025). Mac-sql: A multi-agent collaborative framework for text-to-sql. In Proceedings of the 31st International Conference on Computational Linguistics, pages 540–557.

Wang, B., Shin, R., Liu, X., Polozov, O., and Richardson, M. (2020). Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 7567–7578.

Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911–3921.

Zhong, R., Yu, T., and Klein, D. (2020). Semantic evaluation for text-to-SQL with distilled test suites. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396–411, Online. Association for Computational Linguistics.