LLMs para Detecção de Phishing em E-mails: Comparação Controlada com Baselines TF-IDF e Red Flags Auditáveis
Resumo
Este trabalho avalia Grandes Modelos de Linguagem, do inglês, Large Language Models (LLMs) na detecção de phishing em e-mails sob um protocolo reprodutível, controlado e orientado à cibersegurança. Comparamos quatro LLMs recentes em um mesmo conjunto avaliativo de 90 mensagens, separado de uma etapa técnica de calibração com 10 mensagens, e confrontamos os resultados com baselines TF-IDF treinadas sem vazamento do conjunto de avaliação. Entre os LLMs, o Qwen3 32B obteve o melhor desempenho e revocação perfeita para phishing, enquanto o Llama 3.3 70B apresentou o perfil mais equilibrado entre precisão e revocação. Entretanto, as baselines TF-IDF alcançaram maior F1-score bruto neste recorte textual. A contribuição, portanto, não é afirmar superioridade geral dos LLMs, mas analisar quando eles agregam valor operacional por meio de red flags auditáveis e justificativas estruturadas.
Referências
Anti-Phishing Working Group (2025). Phishing activity trends report: 4th quarter 2024. Technical report, APWG. Released March 19, 2025.
Bergholz, A., De Beer, J., Glahn, S., Moens, M.-F., Paass, G., and Strobel, S. (2010). New filtering approaches for phishing email. Journal of Computer Security, 18(1):7–35.
Bustamante, E. E., Rocha, A. M., Quincozes, S. E., Kazienko, J. F., and Quincozes, V. E. (2025). Caracterização de phishing com grandes modelos de linguagem (llms): Uma avaliação comparativa entre gemini, deepseek e chatgpt. In Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais (SBSeg), pages 204–215. SBC.
Chataut, R., Gyawali, P. K., and Usman, Y. (2024). Can ai keep you safe? a study of large language models for phishing detection. In 2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC), pages 548–554. IEEE. [link]
Fette, I., Sadeh, N., and Tomasic, A. (2007). Learning to detect phishing emails. In Proceedings of the 16th International Conference on World Wide Web, pages 649–656. ACM.
Heiding, F., Schneier, B., Vishwanath, A., Bernstein, J., and Park, P. S. (2024). Devising and detecting phishing emails using large language models. IEEE Access, 12:42131–42146.
Jakobsson, M. and Myers, S., editors (2006). Phishing and Countermeasures: Understanding the Increasing Problem of Electronic Identity Theft. Wiley-Interscience.
Koide, T., Fukushi, N., Nakano, H., and Chiba, D. (2024a). Chatspamdetector: Leveraging large language models for effective phishing email detection. CoRR abs/2402.18093.
Koide, T., Nakano, H., and Chiba, D. (2024b). ChatPhishDetector: Detecting phishing sites using large language models. IEEE Access, 12:154381–154400.
Li, Y., Huang, C., Deng, S., Lock, M. L., Cao, T., Oo, N., Lim, H. W., and Hooi, B. (2024). KnowPhish: Large language models meet multimodal knowledge graphs for enhancing Reference-Based phishing detection. In 33rd USENIX Security Symposium (USENIX Security 24), pages 793–810, Philadelphia, PA. USENIX Association.
Lim, B., Huerta, R., Sotelo, A., Quintela, A., and Kumar, P. (2025). Explicate: Enhancing phishing detection through explainable ai and llm-powered interpretability. arXiv preprint arXiv:2503.20796.
Lundberg, S. M. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30.
Mahendru, S. and Pandit, T. (2024). Securenet: A comparative study of deberta and large language models for phishing detection. In 2024 IEEE 7th International Conference on Big Data and Artificial Intelligence (BDAI), pages 160–169. IEEE.
Nahmias, D., Engelberg, G., Klein, D., and Shabtai, A. (2024). Prompted contextual vectors for spear-phishing detection. CoRR abs/2402.08309.
Nair, R., Abbasi, F. H., and Pervez, S. (2025). Phishemailllm: A meta model approach to detect phishing emails by leveraging llms and machine learning models. In Proceedings of the 2025 Australasian Computer Science Week (ACSW 2025), pages 19–29. ACM.
Quincozes, C. B., Molinos, D., Araújo, R. D., and Quincozes, S. E. (2025). Indicadores semânticos na engenharia de prompts: Uma abordagem explicável para detecçao de fake news. In Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais (SBSeg), pages 1113–1120. SBC.
Salton, G. and Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513–523.
Sharevski, F., Devine, A., Pieroni, E., and Jachim, P. (2022). Phishing with malicious qr codes. In Proceedings of the 2022 European Symposium on Usable Security, pages 160–171. ACM.
Sheng, S., Holbrook, M., Kumaraguru, P., Cranor, L. F., and Downs, J. (2010). Who falls for phish? a demographic analysis of phishing susceptibility and effectiveness of interventions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 373–382. ACM.
Subhajournal (2023). Phishing emails dataset. Kaggle dataset. Acesso em 11 maio 2026.
Uddin, M. A. and Sarker, I. H. (2024). An explainable transformer-based model for phishing email detection: A large language model approach. CoRR abs/2402.13871.
Bergholz, A., De Beer, J., Glahn, S., Moens, M.-F., Paass, G., and Strobel, S. (2010). New filtering approaches for phishing email. Journal of Computer Security, 18(1):7–35.
Bustamante, E. E., Rocha, A. M., Quincozes, S. E., Kazienko, J. F., and Quincozes, V. E. (2025). Caracterização de phishing com grandes modelos de linguagem (llms): Uma avaliação comparativa entre gemini, deepseek e chatgpt. In Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais (SBSeg), pages 204–215. SBC.
Chataut, R., Gyawali, P. K., and Usman, Y. (2024). Can ai keep you safe? a study of large language models for phishing detection. In 2024 IEEE 14th Annual Computing and Communication Workshop and Conference (CCWC), pages 548–554. IEEE. [link]
Fette, I., Sadeh, N., and Tomasic, A. (2007). Learning to detect phishing emails. In Proceedings of the 16th International Conference on World Wide Web, pages 649–656. ACM.
Heiding, F., Schneier, B., Vishwanath, A., Bernstein, J., and Park, P. S. (2024). Devising and detecting phishing emails using large language models. IEEE Access, 12:42131–42146.
Jakobsson, M. and Myers, S., editors (2006). Phishing and Countermeasures: Understanding the Increasing Problem of Electronic Identity Theft. Wiley-Interscience.
Koide, T., Fukushi, N., Nakano, H., and Chiba, D. (2024a). Chatspamdetector: Leveraging large language models for effective phishing email detection. CoRR abs/2402.18093.
Koide, T., Nakano, H., and Chiba, D. (2024b). ChatPhishDetector: Detecting phishing sites using large language models. IEEE Access, 12:154381–154400.
Li, Y., Huang, C., Deng, S., Lock, M. L., Cao, T., Oo, N., Lim, H. W., and Hooi, B. (2024). KnowPhish: Large language models meet multimodal knowledge graphs for enhancing Reference-Based phishing detection. In 33rd USENIX Security Symposium (USENIX Security 24), pages 793–810, Philadelphia, PA. USENIX Association.
Lim, B., Huerta, R., Sotelo, A., Quintela, A., and Kumar, P. (2025). Explicate: Enhancing phishing detection through explainable ai and llm-powered interpretability. arXiv preprint arXiv:2503.20796.
Lundberg, S. M. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30.
Mahendru, S. and Pandit, T. (2024). Securenet: A comparative study of deberta and large language models for phishing detection. In 2024 IEEE 7th International Conference on Big Data and Artificial Intelligence (BDAI), pages 160–169. IEEE.
Nahmias, D., Engelberg, G., Klein, D., and Shabtai, A. (2024). Prompted contextual vectors for spear-phishing detection. CoRR abs/2402.08309.
Nair, R., Abbasi, F. H., and Pervez, S. (2025). Phishemailllm: A meta model approach to detect phishing emails by leveraging llms and machine learning models. In Proceedings of the 2025 Australasian Computer Science Week (ACSW 2025), pages 19–29. ACM.
Quincozes, C. B., Molinos, D., Araújo, R. D., and Quincozes, S. E. (2025). Indicadores semânticos na engenharia de prompts: Uma abordagem explicável para detecçao de fake news. In Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais (SBSeg), pages 1113–1120. SBC.
Salton, G. and Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513–523.
Sharevski, F., Devine, A., Pieroni, E., and Jachim, P. (2022). Phishing with malicious qr codes. In Proceedings of the 2022 European Symposium on Usable Security, pages 160–171. ACM.
Sheng, S., Holbrook, M., Kumaraguru, P., Cranor, L. F., and Downs, J. (2010). Who falls for phish? a demographic analysis of phishing susceptibility and effectiveness of interventions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 373–382. ACM.
Subhajournal (2023). Phishing emails dataset. Kaggle dataset. Acesso em 11 maio 2026.
Uddin, M. A. and Sarker, I. H. (2024). An explainable transformer-based model for phishing email detection: A large language model approach. CoRR abs/2402.13871.
Publicado
01/09/2026
Como Citar
DORNELES, Bernardo; QUINCOZES, Silvio.
LLMs para Detecção de Phishing em E-mails: Comparação Controlada com Baselines TF-IDF e Red Flags Auditáveis. In: WORKSHOP DE TRABALHOS DE INICIAÇÃO CIENTÍFICA E DE GRADUAÇÃO - SIMPÓSIO BRASILEIRO DE CIBERSEGURANÇA (SBSEG), 26. , 2026, Armação dos Búzios/RJ.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 578-589.
DOI: https://doi.org/10.5753/sbseg_estendido.2026.29777.
