How Well Do Small Language Models Understand the Brazilian Educational Context? An Evaluation on ENEM, FUVEST and Unicamp

• Giovani Mambrim Leme Unicamp
• Matheus Ferracciú Scatolin Unicamp
• Anderson Luis Bento Soares Unicamp
• Leonardo Nascimento Ferreira Unicamp
• Ruben Interian Unicamp
• Zanoni Dias Unicamp

DOI: https://doi.org/10.5753/eniac.2026.31313

Resumo

We present a benchmark for Small Language Models (SLMs) on Brazilian high-stakes exams, a setting combining Portuguese reasoning, educational knowledge, and image-based questions rarely covered in prior work. Unlike earlier benchmarks centered on English or native multimodal models, we replace visual items with rich textual descriptions to enable a unified zero-shot evaluation of 12 models on multidisciplinary questions. Our results show that compact models are much cheaper than larger LLMs, but their performance generally remains uneven across subjects and visual-context demands. These findings highlight both the promise and the current limits of SLMs for scalable educational evaluation in Brazilian Portuguese.

Palavras-chave: Small Language Models, Educational Context, Zero-shot Evaluation, Portuguese, LLM Evaluation

Referências

Almeida, T., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabiá-2: A new generation of Portuguese large language models. arXiv preprint arXiv:2403.09887.

Almeida, T. S., Laitz, T., Bonás, G. K., and Nogueira, R. (2023). BLUEX: A benchmark based on Brazilian leading universities entrance exams. In Proceedings of the 12th Brazilian Conference on Intelligent Systems (BRACIS), pages 72–87. SBC.

COMVEST (2026). Comissão permanente para os vestibulares da unicamp. Accessed: 2026-02-10.

Das, R., Hristov, S., Li, H., et al. (2024). Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 7768–7791.

FUVEST (2026). Fundação universitária para o vestibular. Accessed: 2026-02-10.

Godoy, H. (2025). Alvorada-bench: Can language models solve brazilian university entrance exams? arXiv preprint arXiv:2508.15835.

INEP (2026). Exame nacional do ensino médio (enem). Accessed: 2026-02-10.

Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.

Lu, Y., Li, H., Cong, X., et al. (2025). Learning to generate structured output with schema reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 4905–4918.

Nguyen, C., Shen, X., Aponte, R., et al. (2025). A survey on small language models. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing, pages 807–821.

Nunes, D., Primi, R., Pires, R., Lotufo, R., and Nogueira, R. (2023). Evaluating gpt-3.5 and gpt-4 models on brazilian university admission exams. arXiv preprint arXiv:2303.17003.

Pham, N., Kieu, T., Nguyen, D., et al. (2025). Slm-bench: A comprehensive benchmark of small language models on environmental impacts. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 21369–21392.

Pires, R., Abonizio, H., Almeida, T. S., and Nogueira, R. (2023). Sabiá: Portuguese large language models. In Proceedings of the 12th Brazilian Conference on Intelligent Systems (BRACIS), pages 243–258. SBC.

Silveira, I. C. and Mauá, D. D. (2017). University entrance exam as a guiding test for artificial intelligence. In 2017 IEEE Brazilian Conference on Intelligent Systems (BRACIS), pages 426–431. IEEE.