NoobGPT: LLMs and the generation of undetectable malware

• Gustavo Lofrese Carvalho IFC
• Ricardo de la Rocha Ladeira IFC
• Gabriel Eduardo Lima UFPR

DOI: https://doi.org/10.5753/wsis.2025.15064

Abstract

This paper explores the ability of the ChatGPT language model to generate malware through instructions known as jailbreaks, which are widely shared on the internet. The research simulates the use by a lay user with no cybersecurity skills and evaluates the malicious code generated in terms of its functionality and detection by security software. The tests showed that it is possible to obtain different types of malware, most of which were undetectable in the first interactions. It was also observed that the security mechanisms, although present, can be bypassed. The results raise concerns about the misuse of LLMs and the current limitations of digital protection tools.

References

Gupta, M., Akiri, C., Aryal, K., Parker, E., e Praharaj, L. (2023). From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy. IEEE Access, 11, pp. 80218–80245.

Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., Wang, K., & Liu, Y. (2024). Jailbreaking ChatGPT via prompt engineering: An empirical study (arXiv:2305.13860). arXiv.

Madani, P. (2023). Metamorphic malware evolution: The potential and peril of large language models. In: 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), pp. 74–81.

OpenAI. (2025). Using GPTs on our Free Tier (FAQ). [link].

OpenAI Community. (2025). Interaction limits for ChatGPT-4. [link].

Pa, Y. M., Tanizaki, S., Kou, T., Van Eeten, M., Yoshioka, K., e Matsumoto, T. (2023). An attacker’s dream? Exploring the capabilities of ChatGPT for developing malware. In: Proceedings of the 16th Cyber Security Experimentation and Test Workshop.

Stanford University. (2024). The 2024 AI Index Report. [link].

Tahir, R. (2018). A study on malware and malware detection techniques. International Journal of Education and Management Engineering, 8(2), p. 20. Modern Education and Computer Science Press.

Xu, Z., Liu, Y., Deng, G., Li, Y., & Picek, S. (2024). A comprehensive study of jailbreak attack versus defense for large language models. In: Findings of the Association for Computational Linguistics: ACL 2024 (pp. 7432–7449).

Yamin, M. M., Hashmi, E., e Katt, B. (2024). Combining uncensored and censored LLMs for ransomware generation. In: International Conference on Web Information Systems Engineering, pp. 189–202. Springer.

Yong, Z. X., Menghini, C., & Bach, S. (2023). Low-resource languages jailbreak GPT-4. In: Socially Responsible Language Modelling Research.

Yong Wong, M., Landen, M., Antonakakis, M., Blough, D. M., Redmiles, E. M., e Ahamad, M. (2021). An inside look into the practice of malware analysis. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 3053–3069.

Zhu, K. (2024). Ranked: The most popular generative AI tools in 2024. Visual Capitalist. [link].