Hands-On LLM Security: A Classroom Experiment

• Ricardo de la R. Ladeira IFC
• Lucas A. C. Marin IFC
• João V. Espig IFC
• Mickael Reichert IFC
• Carlos E. P. Fidélis IFC
• André V. B. de Macêdo IFC
• Andrey M. Friedel IFC
• Bernardo de L. Machado IFC
• Eliézer da Silva IFC
• Guilherme de Campos IFC
• Leanderson L. de S. Nascimento IFC
• Mateus A. Dall’igna IFC
• Ygor Antunes IFC
• Guilherme H. dos Santos IFC

DOI: https://doi.org/10.5753/errc.2026.36075

Resumo

This paper presents a hands-on classroom activity exploring Large Language Model security vulnerabilities in an undergraduate Cybersecurity course. Students tested aligned models against attack vectors, including prompt injection and jailbreaking. Results show that models could be manipulated to generate unsafe outputs such as functional malware and prohibited instructions, highlighting critical safety gaps. Learning outcomes show a 84.6% correct-response rate and positive self-reported confidence and interest in analyzing AI risks, indicating the value of practical vulnerability exploration.

Referências

Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI feedback, arXiv.

Bakulin, I. et al. (2025). FLAME: Flexible LLM-Assisted Moderation Engine, arXiv.

Bommasani, R. et al. (2021). On the opportunities and risks of foundation models, arXiv.

Chen, S.-W. et al. (2025). Hands-On Training Framework for Prompt Injection Exploits in Large Language Models. Engineering Proceedings, 108(1), 25.

Devadiga, D. J. et al. (2026). A virtual lab for learning AI security and adversarial prompt engineering. ITiCSE’26, 107-113.

Fidelis, C. E. P., & Ladeira, R. R. (2026). Injeção Indireta de Prompt no Gemini. Anais do Computer on the Beach, 17, 689-691.

Gupta, M., et al. (2023). From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy. IEEE Access, 11, 80218–80245.

Huang, S. et al. (2024). Collective constitutional AI: Aligning a language model with public input. ACM FAccT.

Kosiur, D. (2001). Understanding Policy-Based Networking (2 ed.). Wiley.

Liu, Y. et al. (2024). Jailbreaking ChatGPT via prompt engineering: An empirical study, arXiv.

Microsoft. (2024). Best practices to architect secure generative AI applications.

Ministério da Educação (2016). Resolução CNE/CES nº 5, de 16 de novembro de 2016.

Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 35, 27730-27744.

OWASP. (2025a). LLM01: Prompt Injection. OWASP Foundation.

OWASP. (2025b). LLM04: Model Denial of Service. OWASP Foundation.

Rawte, V. et al. (2023). A survey of hallucination in large foundation models, arXiv.

Wilson, D. A. (2026). Integrating Adversarial Scenarios into LLM Security Labs: An Experience Report on a Hands-On Approach. JCERP, 2026(1), 2.

Xu, Z. et al. (2024). A comprehensive study of jailbreak attack versus defense for large language models. ACL 2024, 7432–7449.

Yong, Z. et al. (2023). Low-resource languages jailbreak GPT-4. SoLaR Workshop.

Zhang, Y., Carlini, N. & Ippolito, D. (2024). Effective prompt extraction from language models. COLM’24.