On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTs

• Gefté Almeida UNIPAMPA
• Marcio Pohlmann UNIPAMPA
• Alex Severo UNIPAMPA
• Diego Kreutz UNIPAMPA
• Tiago Heinrich MPI
• Lourenço Pereira ITA

DOI: https://doi.org/10.5753/errc.2025.17811

Resumo

In this study, we evaluate open-source models for security incident classification, comparing them with proprietary models. We utilize a dataset of anonymized real incidents, categorized according to the NIST SP 800-61r3 taxonomy and processed using five prompt-engineering techniques (PHP, SHP, HTP, PRP, and ZSL). The results indicate that, although proprietary models still exhibit higher accuracy, locally deployed open-source models provide advantages in privacy, cost-effectiveness, and data sovereignty.

Referências

Irugalbandara, C., Mahendra, A., Daynauth, R., Arachchige, T. K., Dantanarayana, J., Flautner, K., Tang, L., Kang, Y., and Mars, J. (2024). Scaling down to scale up: A cost-benefit analysis of replacing openai’s llm with open source slms in production. arXiv preprint arXiv:2312.14972.

Kassianik, P., Saglam, B., Chen, A., Nelson, B., Vellore, A., Aufiero, M., Burch, F., Kedia, D., Zohary, A., Weerawardhena, S., Priyanshu, A., Swanda, A., Chang, A., Anderson, H., Oshiba, K., Santos, O., Singer, Y., and Karbasi, A. (2025). Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report. Technical Report.

Noreika, A. (2025). Open Source vs Proprietary LLMs: The Key Differences. SentiSight (Neurotechnology). Online Article.

Pan, G. and Wang, H. (2025). A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services. Working Paper, Carnegie Mellon University.

Pohlmann, M., Severo, A., Almeida, G., Kreutz, D., Heinrich, T., and Pereira, L. (2025). Temperature in SLMs: Impact on incident categorization in on-premises environments. [link].

Salahuddin, S., Hussain, A., Löppönen, J., Jutila, T., and Papadimitratos, P. (2025). Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens. arXiv preprint arXiv:2412.01633.

Severo, A., Kreutz, D., Bertholdo, L., and Lautert, D. (2025a). Framework de classificação automatizada de incidentes de segurança com llms utilizando técnicas de engenharia de prompt. In Anais do SBSeg.

Severo, A., Lautert, D., Almeida, G., Kreutz, D., Rodrigo, G., Jr, L. P., and Bertholdo, L. (2025b). LLMs e engenharia de prompt para classificação automatizada de incidentes em SOCs. In Anais Estendidos do XXV SBSeg. SBC.

Tellache, A., Korba, A. A., Mokhtari, A., Moldovan, H., and Ghamri-Doudane, Y. (2024). Advancing Autonomous Incident Response: Leveraging LLMs and Cyber Threat Intelligence. Preprint.