Beyond Aggregate Accuracy: Kill Chain-Aware Evaluation of IoMT Intrusion Detection Models - Limeira, Evelin E. D.; Roque, Rafael; Cabral, Luciano; Aires, Fernando; Santos, Wellison R. M.; Lima, Milton; Suruagy, José A. (Anais do Simpósio Brasileiro de Cibersegurança (SBSeg), 2026)
Machine learning-based Intrusion Detection Systems (IDSs) have shown promising results for Internet of Medical Things (IoMT) environments; however, aggregate performance metrics can conceal failures in detecting early-stage behaviors. This paper presents a kill chain-aware analysis of Random Forest, LightGBM, a 1D-CNN, and an autoencoder using the CICIoMT2024 dataset, analyzing detection capability across 18 attack categories and multiple stages of the attack lifecycle. Results show that Random Forest substantially outperforms the CNN in reconnaissance and spoofing scenarios, while default LightGBM configurations exhibit severe degradation when transitioning from binary to multi-class classification. In addition, the benign-only autoencoder achieves high attack recall without requiring attack labels, although its false-positive burden limits direct alert generation. Supported by bootstrap confidence intervals and McNemar’s test, the findings demonstrate that IoMT IDSs should move beyond aggregate accuracy toward phase-aware, class-disaggregated, and statistically grounded assessment protocols.