DataCNPJ: Um Conjunto de Dados Público para Avaliação de Text-to-SQL e Schema-Linking em Inglês e em Português
Resumo
Os principais benchmarks utilizados para validação do desempenho de modelos de linguagem na tarefa de Text-to-SQL, como Spider e BIRD, são limitados ao idioma inglês, não refletem plenamente cenários reais e não oferecem suporte explícito para avaliação da subtarefa de Schema-Linking. O conjunto de dados DataCNPJ, construído a partir de dados públicos do Cadastro Nacional da Pessoa Jurídica (CNPJ), foi projetado para avaliação de modelos nas tarefas de Text-to-SQL e Schema-Linking, oferecendo suporte multilíngue (português e inglês), 187 consultas com diferentes níveis de complexidade e um cenário de validação baseado em dados reais.
Palavras-chave:
Text-to-SQL, Schema-Linking, Modelos de Linguagem, Benchmark Multilíngue
Referências
Floratou, A., Psallidas, F., Zhao, F., Deep, S., Hagleither, G., Tan, W., Cahoon, J., Alotaibi, R., Henkel, J., Singla, A., Grootel, A. V., Chow, B., Deng, K., Lin, K., Campos, M., Emani, K. V., Pandit, V., Shnayder, V., Wang, W., and Curino, C. (2024). NL2SQL is a solved problem... Not! In Conference on Innovative Data Systems Research.
Gan, Y., Chen, X., Huang, Q., Purver, M., Woodward, J. R., Xie, J., and Huang, P. (2021a). Towards Robustness of Text-to-SQL Models against Synonym Substitution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2505–2515, Online. Association for Computational Linguistics.
Gan, Y., Chen, X., and Purver, M. (2021b). Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8926–8931, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., and Huang, X. (2024). Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL. ArXiv, abs/2406.08426.
Lei, F., Chen, J., Ye, Y., Cao, R., Shin, D., Su, H., Suo, Z., Gao, H., Hu, W., Yin, P., Zhong, V., Xiong, C., Sun, R., Liu, Q., Wang, S., and Yu, T. (2025). Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. ArXiv, abs/2411.07763.
Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., Huo, N., Zhou, X., Ma, C., Li, G., Chang, K. C., Huang, F., Cheng, R., and Li, Y. (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Data-base Grounded Text-to-SQLs. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
Liu, X., Shen, S., Li, B., Ma, P., Jiang, R., Zhang, Y., Fan, J., Li, G., Tang, N., and Luo, Y. (2024). A Survey of NL2SQL with Large Language Models: Where are we, and where are we going? ArXiv, abs/2408.05109.
Lopes, A., Nogueira, R., Lotufo, R., and Pedrini, H. (2020). Lite Training Strategies for Portuguese-English and English-Portuguese Translation. In Proceedings of the Fifth Conference on Machine Translation, pages 833–840, Online. Association for Computational Linguistics.
May, W. (1999). Information Extraction and Integration with FLORID: The MONDIAL Case Study. Technical Report 131, Universität Freiburg, Institut für Informatik. Available from [link].
Oliveira, A., Nascimento, E., Pinheiro, J. a., Avila, C. V. S., Coelho, G., Feijó, L., Izquierdo, Y., García, G., Leme, L. A. P. P., Lemos, M., and Casanova, M. A. (2024). Small, Medium, and Large Language Models for Text-to-SQL. In Conceptual Modeling: 43rd International Conference, ER 2024, Pittsburgh, PA, USA, October 28–31, 2024, Proceedings, page 276–294, Berlin, Heidelberg. Springer-Verlag.
Pourreza, M. and Rafiei, D. (2024). DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8212–8220, Miami, Florida, USA. Association for Computational Linguistics.
Silva, L., Silva, P. H., and Silva, F. (2025a). Leis de Escala para Text-to-SQL: Um Estudo sobre a Relação entre Tamanho e Desempenho de Modelos de Linguagem. In Anais do XL Simpósio Brasileiro de Bancos de Dados, pages 140–153, Porto Alegre, RS, Brasil. SBC.
Silva, P., Silva, L., and Silva, F. (2025b). Towards Small Language Models for Text-to-SQL. In Anais da XXXV Brazilian Conference on Intelligent Systems, pages 516–531, Porto Alegre, RS, Brasil. SBC.
Volvovsky, S., Marcassa, M., and Panbiharwala, M. (2024). DFIN-SQL: Integrating Focused Schema with DIN-SQL for Superior Accuracy in Large-Scale Databases. ArXiv, abs/2403.00872.
Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q., Yin, D., Sun, X., and Li, Z. (2025). MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, pages 540–557, Abu Dhabi, UAE. Association for Computational Linguistics.
Yang, S., Su, Q., Li, Z., Li, Z., Mao, H., Liu, C., and Zhao, R. (2024). SQL-to-Schema Enhances Schema Linking in Text-to-SQL. In Database and Expert Systems Applications - 35th International Conference, DEXA 2024.
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. (2018). Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMLP 2018, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
Zhong, R., Yu, T., and Klein, D. (2020). Semantic Evaluation for Text-to-SQL with Distilled Test Suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020.
Zhong, V., Xiong, C., and Socher, R. (2017). Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. ArXiv, abs/1709.00103.
Gan, Y., Chen, X., Huang, Q., Purver, M., Woodward, J. R., Xie, J., and Huang, P. (2021a). Towards Robustness of Text-to-SQL Models against Synonym Substitution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2505–2515, Online. Association for Computational Linguistics.
Gan, Y., Chen, X., and Purver, M. (2021b). Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8926–8931, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., and Huang, X. (2024). Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL. ArXiv, abs/2406.08426.
Lei, F., Chen, J., Ye, Y., Cao, R., Shin, D., Su, H., Suo, Z., Gao, H., Hu, W., Yin, P., Zhong, V., Xiong, C., Sun, R., Liu, Q., Wang, S., and Yu, T. (2025). Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. ArXiv, abs/2411.07763.
Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., Huo, N., Zhou, X., Ma, C., Li, G., Chang, K. C., Huang, F., Cheng, R., and Li, Y. (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Data-base Grounded Text-to-SQLs. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
Liu, X., Shen, S., Li, B., Ma, P., Jiang, R., Zhang, Y., Fan, J., Li, G., Tang, N., and Luo, Y. (2024). A Survey of NL2SQL with Large Language Models: Where are we, and where are we going? ArXiv, abs/2408.05109.
Lopes, A., Nogueira, R., Lotufo, R., and Pedrini, H. (2020). Lite Training Strategies for Portuguese-English and English-Portuguese Translation. In Proceedings of the Fifth Conference on Machine Translation, pages 833–840, Online. Association for Computational Linguistics.
May, W. (1999). Information Extraction and Integration with FLORID: The MONDIAL Case Study. Technical Report 131, Universität Freiburg, Institut für Informatik. Available from [link].
Oliveira, A., Nascimento, E., Pinheiro, J. a., Avila, C. V. S., Coelho, G., Feijó, L., Izquierdo, Y., García, G., Leme, L. A. P. P., Lemos, M., and Casanova, M. A. (2024). Small, Medium, and Large Language Models for Text-to-SQL. In Conceptual Modeling: 43rd International Conference, ER 2024, Pittsburgh, PA, USA, October 28–31, 2024, Proceedings, page 276–294, Berlin, Heidelberg. Springer-Verlag.
Pourreza, M. and Rafiei, D. (2024). DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8212–8220, Miami, Florida, USA. Association for Computational Linguistics.
Silva, L., Silva, P. H., and Silva, F. (2025a). Leis de Escala para Text-to-SQL: Um Estudo sobre a Relação entre Tamanho e Desempenho de Modelos de Linguagem. In Anais do XL Simpósio Brasileiro de Bancos de Dados, pages 140–153, Porto Alegre, RS, Brasil. SBC.
Silva, P., Silva, L., and Silva, F. (2025b). Towards Small Language Models for Text-to-SQL. In Anais da XXXV Brazilian Conference on Intelligent Systems, pages 516–531, Porto Alegre, RS, Brasil. SBC.
Volvovsky, S., Marcassa, M., and Panbiharwala, M. (2024). DFIN-SQL: Integrating Focused Schema with DIN-SQL for Superior Accuracy in Large-Scale Databases. ArXiv, abs/2403.00872.
Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q., Yin, D., Sun, X., and Li, Z. (2025). MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, pages 540–557, Abu Dhabi, UAE. Association for Computational Linguistics.
Yang, S., Su, Q., Li, Z., Li, Z., Mao, H., Liu, C., and Zhao, R. (2024). SQL-to-Schema Enhances Schema Linking in Text-to-SQL. In Database and Expert Systems Applications - 35th International Conference, DEXA 2024.
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. (2018). Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMLP 2018, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
Zhong, R., Yu, T., and Klein, D. (2020). Semantic Evaluation for Text-to-SQL with Distilled Test Suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020.
Zhong, V., Xiong, C., and Socher, R. (2017). Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. ArXiv, abs/1709.00103.
Publicado
08/09/2026
Como Citar
SILVA, Paulo H. C.; SILVA, Letícia O.; ISSA, Érick V.; SILVA, Fabrício A..
DataCNPJ: Um Conjunto de Dados Público para Avaliação de Text-to-SQL e Schema-Linking em Inglês e em Português. In: DATASET SHOWCASE WORKSHOP (DSW), 8. , 2026, São Carlos/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 83-92.
DOI: https://doi.org/10.5753/dsw.2026.249553.
