Malware development using the AIM prompt in ChatGPT

• Gustavo Lofrese Carvalho IFC
• Ricardo de la Rocha Ladeira IFC
• Gabriel Eduardo Lima UFPR

DOI: https://doi.org/10.5753/eries.2025.14922

Abstract

This paper explores ChatGPT’s ability to generate malware through instructions that remove its restrictions known as jailbreaks. The malware was statically analyzed to understand their internal structure and executed in a controlled environment to verify their behavior. The results showed that it is possible to obtain malware through ChatGPT, even with security locks.

References

Gupta, M., Akiri, C., Aryal, K., Parker, E. & Praharaj, L. (2023). From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy. IEEE Access, 11, pp. 80218–80245.

Madani, P. (2023). Metamorphic malware evolution: The potential and peril of large language models. In: 5th IEEE TPS-ISA, pp. 74–81.

OpenAI Community. (2025). Interaction limits for ChatGPT-4. [link].

Stanford University. (2024). The 2024 AI Index Report. [link].

Tahir, R. (2018). A study on malware and malware detection techniques. International Journal of Education and Management Engineering, 8(2), p. 20.

Wong, M. Y., Landen, M., Antonakakis, M., Blough, D. M., Redmiles, E. M. & Ahamad, M. (2021). An inside look into the practice of malware analysis. In: Proceedings of the 2021 ACM CCS, pp. 3053–3069.

Xu, Z., Liu, Y., Deng, G., Li, Y. & Picek, S. (2024). A comprehensive study of jailbreak attack versus defense for large language models. In: Findings of the ACL 2024, pp. 7432–7449.

Yamin, M. M., Hashmi, E., e Katt, B. (2024). Combining uncensored and censored LLMs for ransomware generation. In: WISE 2024, pp. 189–202.

Yong, Z. X., Menghini, C. & Bach, S. (2023). Low-resource languages jailbreak GPT-4. In: Socially Responsible Language Modelling Research.