Autonomous Red Teaming for Large Language Models: A Minimally Supervised Approach Using Specialized Agents - Medeiros, Débora; Vieira, Ana; Dantas, Franklin; Queiroz, Charlene S.; Rodrigues, Marcela; Carvalho, Rebeca V.; Silva Junior, Edson; Ribeiro, Camila; Assumpção, Nicolas R. G.; Carvalho, José R. H. (Anais do Simpósio Brasileiro de Cibersegurança (SBSeg), 2026)
This paper investigates the feasibility of using autonomous AI agents to conduct Red Team evaluations of Large Language Models (LLMs) with minimal human supervision. We propose a framework of specialized agents targeting sensitive domains, including homicide, human trafficking, and zoophilia, which interact with aligned and less aligned LLMs to elicit and identify harmful outputs. The method aligns encouragingly with expert majority judgments, though the low inter-rater agreement highlights the subjectivity of harmfulness assessment, which we discuss as a central limitation. Overall, the framework shows potential as a scalable, cost-efficient support tool for LLM safety evaluation.