Comparative Evaluation of Large Language Models in the Text-to-SQL Task in a Real Domain-Specific Database

• Cesar Yuri e Silva Santos UFRPE
• Samara Monteiro Xavier UFRPE
• Anderson Pinheiro Cavalcanti UFRPE / Cesar School
• Péricles Miranda UFRPE
• Rafael Ferreira Mello UFRPE / Cesar School

DOI: https://doi.org/10.5753/eniac.2026.31429

Resumo

Text-to-SQL busca democratizar o acesso a dados ao traduzir perguntas em linguagem natural para consultas SQL. Contudo, grande parte das avaliações apoia-se em benchmarks acadêmicos padronizados e em métricas automáticas que podem ocultar erros semânticos. Este trabalho apresenta uma avaliação comparativa de cinco LLMs proprietários (Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o, GPT-5 mini e GPT-5.2) sobre um banco de dados PostgreSQL real e de domínio específico, utilizando 22 perguntas em português brasileiro de complexidade variada. Combinamos a acurácia de execução com uma análise de correção semântica das consultas e uma classificação de erros. Os resultados mostram que nenhum modelo domina simultaneamente as duas métricas, que todos têm bom desempenho em consultas simples, mas degradam com o aumento da complexidade.

Referências

Dai, Y., Yang, H., Hao, M., and Chao, P. (2025). Parsql: Enhancing text-to-sql through sql parsing and reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 661–681.

de Carvalho, L. F. C., Júnior, P. S. d. S., and de Oliveira, H. T. A. (2025). Benchmarking large language models for text-to-sql in brazilian portuguese and english. In Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL), pages 101–112. SBC.

Dong, X., Zhang, C., Ge, Y., Mao, Y., Gao, Y., Lin, J., Lou, D., et al. (2023). C3: Zero-shot text-to-sql with chatgpt. arXiv preprint arXiv:2307.07306.

Fan, Y., Weng, Q., Chen, Y., and Wang, X. S. (2025). Rethinking data in nl2sql: a survey of what we have and what we expect. Vicinagearth, 2(1):15.

Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023). Text-to-sql empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363.

Hoang, C. D. V., Vu, D. Q., Tangari, G., Vu, T., Dharmasiri, D., Li, Y.-F., Duong, L., et al. (2025). Sqlong: Enhanced nl2sql for longer contexts with llms. In Proceedings of the 4th Table Representation Learning Workshop, pages 47–55.

Jiang, H., Zhu, M., Zhang, Q., Zhong, G., and Li, G. (2025). Research on nl2sql in the logistics field based on large language models under resource constraints. In Proceedings of the 2025 5th International Conference on Big Data, Artificial Intelligence and Risk Management, pages 317–325.

Kokolishvili, A. (2026). A study of the performance of large language models in text-to-sql tasks. In 2026 5th International Informatics and Software Engineering Conference (IISEC), pages 315–319.

Liu, X., Shen, S., Li, B., Ma, P., Jiang, R., Zhang, Y., Fan, J., Li, G., Tang, N., and Luo, Y. (2025a). A survey of text-to-sql in the era of llms: Where are we, and where are we going? IEEE Transactions on Knowledge and Data Engineering, 37(10):5735–5754.

Liu, X., Shen, S., Li, B., Tang, N., and Luo, Y. (2025b). Nl2sql-bugs: A benchmark for detecting semantic errors in nl2sql translation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5662–5673.

Nobre, L., Sousa, S., Teles, S., and Soares, A. (2026). Language effects in text-to-sql across english and portuguese. In Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026)-Vol. 1, pages 270–280.

Pedroso, B. C., Pereira, M. R., and Pereira, D. A. (2025). Performance evaluation of llms in the text-to-sql task in portuguese. In Simpósio Brasileiro de Sistemas de Informação (SBSI), pages 260–269. SBC.

Pei, W., Xu, H., Zhao, H. H., Hou, S., Han, C., Zhang, Z., Pingyi, L., and He, B. (2025). Feather-sql: A lightweight nl2sql framework with dual-model collaboration paradigm for small language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 2069–2086.

Shi, L., Tang, Z., Zhang, N., Zhang, X., and Yang, Z. (2025). A survey on employing large language models for text-to-sql tasks. ACM Computing Surveys, 58(2):1–37.

Silva, P. H., Silva, L. O., and Silva, F. A. (2025). Towards small language models for text-to-sql. In Brazilian Conference on Intelligent Systems, pages 516–531. Springer.

Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., Sun, X., et al. (2025). Mac-sql: A multi-agent collaborative framework for text-to-sql. In Proceedings of the 31st International Conference on Computational Linguistics, pages 540–557.

Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911–3921.

Zhan, G., Shi, G., Liu, X., and Wang, N. (2025). Natural language to sql using large language models. In 2025 21st International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pages 495–502.

Zhang, T., Chen, C., Liao, C., Wang, J., Zhao, X., Yu, H., Wang, J., Li, J., and Shi, W. (2024). Sqlfuse: Enhancing text-to-sql performance through comprehensive llm synergy. arXiv preprint arXiv:2407.14568.