Benchmarking Large Language Models for Text-to-SQL in Brazilian Portuguese and English - Carvalho, Luís Felippe C. de; S. Júnior, Paulo Sérgio dos; Oliveira, Hilário Tomaz Alves de (Anais do Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL); 2025: Anais do XVI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, 2025)
This work assessed the performance of seventeen large language models, including open-source and proprietary models, on the Text-to-SQL task in both Brazilian Portuguese and English. Two schema representation strategies were considered: textual descriptions and representations using data definition language. Experimental results on the Spider dataset demonstrated the superior performance of proprietary models, particularly the Gemini-2.5-Flash-preview, as measured by the Execution Accuracy and Exact Match metrics. Among the open-source models, Qwen-2.5-Coder-14B achieved the highest performance. An error analysis of the best-performing model revealed strong proficiency in handling clauses such as SELECT and AND/OR, while considerable challenges persisted in generating more complex constructs, including GROUP BY with HAVING and set operators like UNION and INTERSECT.