Beyond Aggregate Accuracy: Kill Chain-Aware Evaluation of IoMT Intrusion Detection Models

• Evelin E. D. Limeira CESAR School
• Rafael Roque CESAR School
• Luciano Cabral CESAR School / IFPE
• Fernando Aires CESAR School / UFRPE
• Wellison R. M. Santos CESAR School / UFRPE
• Milton Lima CESAR School
• José A. Suruagy CESAR School

DOI: https://doi.org/10.5753/sbseg.2026.27104

Resumo

Machine learning-based Intrusion Detection Systems (IDSs) have shown promising results for Internet of Medical Things (IoMT) environments; however, aggregate performance metrics can conceal failures in detecting early-stage behaviors. This paper presents a kill chain-aware analysis of Random Forest, LightGBM, a 1D-CNN, and an autoencoder using the CICIoMT2024 dataset, analyzing detection capability across 18 attack categories and multiple stages of the attack lifecycle. Results show that Random Forest substantially outperforms the CNN in reconnaissance and spoofing scenarios, while default LightGBM configurations exhibit severe degradation when transitioning from binary to multi-class classification. In addition, the benign-only autoencoder achieves high attack recall without requiring attack labels, although its false-positive burden limits direct alert generation. Supported by bootstrap confidence intervals and McNemar’s test, the findings demonstrate that IoMT IDSs should move beyond aggregate accuracy toward phase-aware, class-disaggregated, and statistically grounded assessment protocols.

Referências

Adji, L. H. E., Cunha, L. F. I., and Machado, R. C. S. (2025). Detecção de botnets em dispositivos IoT utilizando análise de consultas DNS com one-class SVM. In Anais do SBSeg 2025: Artigos Completos, pages 1–17. Sociedade Brasileira de Computação.

Akar, G., Sahmoud, S., Onat, M., Cavusoglu, Ü., and Malondo, E. (2025). L2d2: A novel lstm model for multi-class intrusion detection systems in the era of iomt. IEEE Access, 13:7002–7013.

Ayad, A. G., Sakr, N. A., and Hikal, N. A. (2025). Fog-empowered anomaly detection in iot networks using one-class asymmetric stacked autoencoder. Cluster Computing, 28(8).

Breiman, L. (2001). Random forests. Machine Learning, 45(1):5–32.

Chandekar, P. B., Mehta, M. S., and Chandan, S. (2025). Enhanced anomaly detection in iomt networks using ensemble ai models on the ciciomt2024 dataset. arXiv preprint arXiv:2502.11854.

Dadkhah, S., Neto, E. C. P., Ferreira, R., Molokwu, R. C., Sadeghi, S., and Ghorbani, A. A. (2024). Ciciomt2024: A benchmark dataset for multi-protocol security assessment in iomt. Internet of Things, 28:101351.

Damasceno, M. G. L., Souza, C. B. B., and Balieiro, A. M. (2025). Evaluating MLP and autoencoder models for zero-day attack detection in 6G networks. In Anais do SBSeg 2025: Artigos Completos, pages 1–17. Sociedade Brasileira de Computação.

Doménech, J., León, O., Siddiqui, M. S., and Pegueroles, J. (2025). Evaluating and enhancing intrusion detection systems in iomt: the importance of domain-specific datasets. Internet of Things, 32:101631.

Ehrenfeld, J. M. (2017). WannaCry, cybersecurity and health information technology: A time to act. Journal of Medical Systems, 41(7):104.

Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1–22.

Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. Adaptive Computation and Machine Learning. The MIT Press, Cambridge, MA, USA. Chapter 14: Autoencoders.

Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.

Hutchins, E. M., Cloppert, M. J., and Amin, R. M. (2011). Intelligence-driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains. In Proceedings of the 6th International Conference on Information Warfare and Security (ICIW 2011), pages 113–125, Washington, DC, USA. Academic Conferences and Publishing International Ltd. Conference held 17–18 March 2011 at George Washington University; also issued as Lockheed Martin white paper.

Kavkas, N. C. and Yildiz, K. (2025). Enhancing iomt security with deep learning based approach for medical iot threat detection. In IEEE International Symposium on Digital Forensic and Security, ISDFS, pages 1–6.

Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 3146–3154, Long Beach, CA, USA. Curran Associates, Inc.

McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. PMID: 20254758.

MITRE ATT&CK (2025). Enterprise tactics: Reconnaissance and impact. [link]. Accessed: 2026-05-11.

Mohammadi, A., Ghahramani, H., Asghari, S. A., and Aminian, M. (2024). Securing healthcare with deep learning: A cnn-based model for medical iot threat detection. In 19th Iranian Conference on Intelligent Systems, ICIS, pages 168–173.

Ruellan, E., Paquet-Clouston, M., and Garcia, S. (2024). Conti inc.: Understanding the internal discussions of a large ransomware-as-a-service operator with machine learning. Crime Science, 13(16).

Sohail, F., Bhatti, M. A. M., Awais, M., and Iqtidar, A. (2024). Explainable boosting ensemble methods for intrusion detection in internet of medical things (iomt) applications. In International Conference on Digital Transformation, ICoDT2, pages 1–6. IEEE.

Uddin, M. A., Chu, N. H., and Rafeh, R. (2025). A hierarchical ids for zero-day attack detection in internet of medical things networks. arXiv preprint arXiv:2508.10346.

Vinayakumar, R., Alazab, M., Soman, K. P., Poornachandran, P., Al-Nemrat, A., and Venkatraman, S. (2019). Deep learning approach for intelligent intrusion detection system. IEEE Access, 7:41525–41550.

Yacoubi, M., Moussaoui, O., and Drocourt, C. (2026). Enhancing iomt security with explainable machine learning: A case study on the ciciomt2024 dataset. In Connected Objects, Artificial Intelligence, Telecommunications and Electronics Engineering, pages 249–254, Cham. Springer Nature Switzerland.