Evaluating Lightweight Transformers for Network Intrusion Detection under Temporal Drift: A Comparative Study with MAWIFlow

João Jardim Neto; César Augusto Cavalheiro Marcondes

doi:10.5753/sbseg.2026.27067

João Jardim Neto ITA
César Augusto Cavalheiro Marcondes ITA

DOI: https://doi.org/10.5753/sbseg.2026.27067

Resumo

Static train/test splits remain the dominant evaluation protocol for network intrusion detection systems (NIDS), yet real-world traffic evolves continuously and deployed models degrade under temporal drift. We present a reproducible comparative evaluation of four model families—XGBoost, MLP, CNN-BiLSTM, and a parameter-efficient causal Transformer (LightTransformer, a compact FlowTransformer-derived configuration)—on two complementary protocols: a static split on CICIoT2023 (sanity check) and a temporal forward-chaining evaluation on MAWIFlow (2007–2024). On the static benchmark all models achieve F1₊>0.98 on the attack class, so near-ceiling static scores are insufficient evidence of temporal robustness. Under temporal drift (cumulative training window, R=5 seeds, all models) the LightTransformer obtains nAUT₁=0.57 (95% CI [0.52, 0.62]) versus 0.44 (CNN-BiLSTM), 0.38 (XGBoost), and 0.28 (MLP). Bootstrap confidence intervals are consistent with this ordering, which holds across all five seeds; a Friedman test (χ²=15.0, p=0.002) rejects equal rankings globally; one-sided pairwise Wilcoxon signed-rank tests provide uncorrected directional evidence (p=0.031 for all six pairs, n=5); and no pairwise comparison survives Bonferroni correction (α_c=0.0083). Our results indicate that forward-chaining temporal evaluation should be a standard component of deployment-oriented NIDS benchmarking.

Referências

Abdulganiyu, O. H., Tchakoucht, T. A., and Saheed, Y. K. (2023). A systematic literature review for network intrusion detection system (ids). International Journal of Information Security, 22:1125–1162.

Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. (2019). Optuna. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623–2631. ACM.

Apruzzese, G., Pajola, L., and Conti, M. (2022). The cross-evaluation of machine learning-based network intrusion detection systems. IEEE Transactions on Network and Service Management, 19:5152–5169.

Chen, T. and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, volume 13-17-August-2016, pages 785–794. Association for Computing Machinery.

Chinnasamy, R., Subramanian, M., Easwaramoorthy, S. V., and Cho, J. (2025). Deep learning-driven methods for network-based intrusion detection systems: A systematic review. ICT Express, 11:181–215.

Fontugne, R., Borgnat, P., Abry, P., and Fukuda, K. (2010). Mawilab. In Proceedings of the 6th International COnference, pages 1–12. ACM.

Goldschmidt, P. and Chudá, D. (2025). Network intrusion datasets: A survey, limitations, and recommendations. Computers & Security, 156:104510.

Kheddar, H. (2025). Transformers and large language models for efficient intrusion detection systems: A comprehensive survey. Information Fusion, 124:103347.

Lansky, J., Ali, S., Mohammadi, M., Majeed, M. K., Karim, S. H., Rashidi, S., Hosseinzadeh, M., and Rahmani, A. M. (2021). Deep learning-based intrusion detection systems: A systematic review.

Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., and Zhang, G. (2018). Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31:1–1.

Manocchio, L. D., Layeghy, S., Lo, W. W., Kulatilleke, G. K., Sarhan, M., and Portmann, M. (2024). Flowtransformer: A transformer framework for flow-based network intrusion detection systems. Expert Systems with Applications, 241:122564.

Neto, E. C. P., Dadkhah, S., Ferreira, R., Zohourian, A., Lu, R., and Ghorbani, A. A. (2023). Ciciot2023: A real-time dataset and benchmark for large-scale attacks in iot environment. Sensors, 23.

Schraven, J., Windmann, A., and Niggemann, O. (2026). Mawiflow benchmark: Realistic flow-based evaluation for network intrusion detection. In Proceedings of the 12th International Conference on Information Systems Security and Privacy, pages 549–556. SCITEPRESS - Science and Technology Publications.

Sommer, R. and Paxson, V. (2010). Outside the closed world: On using machine learning for network intrusion detection. In 2010 IEEE Symposium on Security and Privacy, pages 305–316. IEEE.

Wang, W., Zhu, M., Wang, J., Zeng, X., and Yang, Z. (2017). End-to-end encrypted traffic classification with one-dimensional convolution neural networks. In 2017 IEEE International Conference on Intelligence and Security Informatics (ISI), pages 43–48. IEEE.

Wu, Z., Zhang, H., Wang, P., and Sun, Z. (2022). Rtids: A robust transformer-based approach for intrusion detection system. IEEE Access, 10:64375–64387.

Yin, C., Zhu, Y., Fei, J., and He, X. (2017). A deep learning approach for intrusion detection using recurrent neural networks. IEEE Access, 5:21954–21961.

Zhou, H., Chen, J., Mei, Y., Adam, G., Aggarwal, V., Bastian, N. D., and Lan, T. (2024). Real-time network intrusion detection via importance sampled decision transformers. In Proceedings - 2024 IEEE 21st International Conference on Mobile Ad-Hoc and Smart Systems, MASS 2024, pages 82–91. Institute of Electrical and Electronics Engineers Inc.

Çağdaş Özer and Orman, Z. (2024). Transformers Architecture Oriented Intrusion Detection Systems: A Systematic Review, volume 1138 LNNS, pages 151–160. Springer Science and Business Media Deutschland GmbH.