Evaluating the Robustness of Small Language Models in Portuguese under Red Teaming with Linguistic Perturbations - Bunn, Carlos Daniel S.; Sá, Matheus A.; Pagliuso Neto, Milton P.; Miers, Charles C.; Simplicio Jr., Marcos A. (Anais Estendidos do Simpósio Brasileiro de Cibersegurança (SBSeg); 2026: Anais Estendidos do XXVI Simpósio Brasileiro de CibersegurançaDO - 10.5753/sbseg_estendido.2026.33816, 2026)
Small Language Models are increasingly deployed in resource-constrained settings, yet their safety behavior under linguistic perturbations remains underexplored for Brazilian Portuguese. This work evaluates three open-box Small Language Models (SLMs), Llama 3.2 3B Instruct, Qwen 2.5 3B Instruct, and TucanoBR Tucano-2b4, against 480 adversarial prompts spanning six risk categories, each subjected to five character-level perturbations at three intensity levels, totaling 1,440 model runs. An automated pipeline, using Phi-4-mini-instruct as judge model, measures Objective Satisfaction Rate (OSR), Unsafe Compliance Rate (UCR), Intent Understanding Rate (IUR), and Noise Failure Rate (NFR) to separate genuine safety from refusals caused by incomprehension. Overall unsafe compliance was low across all models (0.42% – 2.71%), with TucanoBR exhibiting the highest UCR despite strong intent understanding, while heavy corruption degraded comprehension rather than eliciting unsafe outputs. Our findings indicate that literal pattern-matching filters are insufficient safeguards against surface-level adversarial evasion.