Comparative Analysis of Automated Red Teaming in LLMs: AttnGCG, GPTFuzzer, and PAPILLON - Martignago, Giovanni Mandel; Pagliuso Neto, Milton Pedro; Miers, Charles Christian (Anais Estendidos do Simpósio Brasileiro de Cibersegurança (SBSeg); 2026: Anais Estendidos do XXVI Simpósio Brasileiro de CibersegurançaDO - 10.5753/sbseg_estendido.2026.29282, 2026)
Automated red-teaming methods enable systematic adversarial testing of Large Language Models, yet direct comparisons across different attack paradigms remain scarce, particularly regarding the trade-off between attack effectiveness and operational cost. We present a controlled comparison of three frameworks spanning distinct strategies: (i) AttnGCG; (ii) GPTFuzzer; and (iii) PAPILLON. All three are evaluated against Llama 3.1 8B Instruct on 18 adversarial goals drawn from HarmBench, using LlamaGuard-3 as a unified judge and an equal budget of 500 iterations per goal. Our results suggest seed-pool-based mutation offers the most favorable cost-effectiveness ratio against the evaluated target model, while gradient-based suffix optimization alone proves largely ineffective against recent safety-aligned architectures within practical iteration budgets. Although PAPILLON is designed to overcome the dependence on pre-existing templates, this advantage comes at a significantly higher operational cost compared to GPTFuzzer, since each mutation is tailored to a specific goal rather than shared across all goals simultaneously.