Small vs. Large Language Models: A Comparative Study on Multiple-Choice Question Answering in Portuguese - Silveira, Marilia Rosa; Lima, Guilherme Dallmann; Silveira Júnior, Carlos Augusto Calage; Freitas, Larissa Astrogildo; Corrêa, Ulisses Brisolara (Proceedings of the Brazilian Symposium on Multimedia and the Web (WebMedia), 2025)
Generative models are widely used for Multiple-Choice Question Answering (MCQA). While performance often improves with model size, prior work reports inconsistencies depending on task, prompting strategy, and language. We evaluate eleven open models ranging from millions to billions of parameters, both monolingual and multilingual, on Portuguese MCQA built from college entrance exams, under six prompting strategies: zero-shot, one-shot, few-shot, shuffled-order (to probe positional effects), and two per-option label-only settings. We also quantify positional bias using a normalized positional-bias coefficient (BPC). Overall, performance increases with parameter count; however, the magnitude varies across strategies. LLaMA-3.1-Storm-8B achieves the best average accuracy, and Sabiá-7B, a model trained with a strong focus on Portuguese, performs competitively among models of similar size. Smaller models (e.g., Tucano-2B, Qwen2-0.5B) attain solid results in specific settings, particularly with per-option scoring. These findings suggest that, although larger models are generally more robust, carefully chosen prompting can make smaller models viable under resource constraints. In summary, performance scales with size but depends on prompting—per-option configurations reduce the SLM–LLM gap, and positional bias is measurable via BPC; future work includes multi-shuffle BPC estimation, calibration, and log-likelihood baselines for per-option scoring and extensions to additional domains and languages.