Evaluation of stateful IDS generalization using machine learning

• Marcelo Fernandes Domingues ITA / Brazilian Navy
• Gustavo de C. Bertoli ITA https://orcid.org/0000-0003-1940-8295
• Leonardo H. de Melo ITA
• Osamu Saotome ITA https://orcid.org/0000-0002-1568-9299
• Aldri Santos UFMG https://orcid.org/0000-0002-5861-4414
• Lourenço Alves Pereira ITA

DOI: https://doi.org/10.5753/sbseg.2022.225165

Abstract

Machine learning is relevant for the characterization of attacks on computer networks, as it allows the identification of traffic patterns and, therefore, the implementation of mechanisms for blocking malicious actions. However, the capacity of solutions to generalize in exogenous contexts is still missing. Hence, we evaluated the performance of different models, sush as DT, LR, MLP, NB, SVM, and XGB, in the UNSW-NB15, CICIDS-2017, BoT-IoT, ToN-IoT, and AB-TRAP datasets. As a result, we observed low levels of generalization in the models. Furthermore, feature engineering enables the comparison of models and leverages the learning process. Finally, we analyze the effectiveness of attributes as predictors of scanning.

Keywords: intrusion detection, machine learning, generalization, stateful

References

Al-Riyami, S., Coenen, F., and Lisitsa, A. (2018). A re-evaluation of intrusion detection accuracy: alternative evaluation strategy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 2195–2197.

Al-Sarawi, S., Anbar, M., Abdullah, R., and Al Hawari, A. B. (2020). Internet of things market analysis forecasts, 2020–2030. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pages 449–453.

Alsaedi, A., Moustafa, N., Tari, Z., Mahmood, A., and Anwar, A. (2020). Ton iot telemetry dataset: A new generation dataset of iot and iiot for data-driven intrusion detection systems. IEEE Access, 8:165130–165150.

Apruzzese, G., Pajola, L., and Conti, M. (2022). The cross-evaluation of machine learning-based network intrusion detection systems. IEEE Transactions on Network and Service Management, pages 1–1.

Bochie, K., Gonzalez, E. R., Giserman, L. F., Campista, M. E. M., and Costa, L.H. M. (2020). Detecção de ataques a redes iot usando técnicas de aprendizado de máquina e aprendizado profundo. In Anais do XX Simpósio Brasileiro em Segurança da Informação e de Sistemas Computacionais. SBC.

Buczak, A. L. and Guven, E. (2016). A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys Tutorials, 18(2):1153–1176.

Catillo, M., Pecchia, A., Rak, M., and Villano, U. (2021). Demystifying the role of public intrusion datasets: a replication study of dos network traffic data. Computers & Security, page 102341.

de Carvalho Bertoli, G., Júnior, L. A. P., Verri, F. A. N., dos Santos, A. L., and Saotome, O. (2021a). Bridging the gap to real-world for network intrusion detection systems with data-centric approach. NeurIPS - Data-centric AI Workshop.

de Carvalho Bertoli, G., Pereira Junior, L. A., Saotome, O., Dos Santos, A. L., Verri, F. A. N., Marcondes, C. A. C., Barbieri, S., Rodrigues, M. S., and Parente De Oliveira, J. M. (2021b). An end-to-end framework for machine learning-based network intrusion detection system. IEEE Access, 9:106790–106805.

Ferrag, M. A., Shu, L., Djallel, H., and Choo, K.-K. R. (2021). Deep learning-based intrusion detection for distributed denial of service attack in agriculture 4.0. Electronics, 10(11).14

Fontugne, R., Borgnat, P., Abry, P., and Fukuda, K. (2010). MAWILab: Combining Diverse Anomaly Detectors for Automated Anomaly Labeling and Performance Benchmarking. In ACM CoNEXT ’10, Philadelphia, PA.

Gupta, L., Salman, T., Ghubaish, A., Unal, D., Al-Ali, A. K., and Jain, R. (2022). Cybersecurity of multi-cloud healthcare systems: A hierarchical deep learning approach. Applied Soft Computing, page 108439.

Habibi Lashkari., A., Draper Gil., G., Mamun., M. S. I., and Ghorbani., A. A. (2017). Characterization of tor traffic using time based features. In Proceedings of the 3rd International Conference on Information Systems Security and Privacy - ICISSP,, pages 253–262. INSTICC, SciTePress.

Kasongo, S. M. and Sun, Y. (2020). Performance analysis of intrusion detection systems using a feature selection method on the unsw-nb15 dataset. Journal of Big Data, 7(1):1–20.

Kenyon, A., Deka, L., and Elizondo, D. (2020). Are public intrusion datasets fit for purpose characterising the state of the art in intrusion event datasets. Computers & Security, 99:102022.

Koroniotis, N., Moustafa, N., Sitnikova, E., and Turnbull, B. (2019). Towards the development of realistic botnet dataset in the internet of things for network forensic analytics: Bot-iot dataset. Future Generation Computer Systems, 100:779–796.

Layeghy, S. and Portmann, M. (2022). On generalisability of machine learning-based network intrusion detection systems.

Lucas, T. J., da Costa, K. A., Moraes, E. A., Júnior, P. R. H., and das Neves, M. J. (2021). Stacking-based committees para detecção de ataques em redes de computadores-uma abordagem por exaustão. In Anais do XXXIX Simpósio Brasileiro de Redes de Computadores e Sistemas Distribu??dos, pages 644–657.

Moustafa, N. and Slay, J. (2015). Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In 2015 military communications and information systems conference (MilCIS), pages 1–6. IEEE.

Roy, S., Li, J., Choi, B.-J., and Bai, Y. (2022). A lightweight supervised intrusion detection mechanism for iot networks. Future Generation Computer Systems, 127.

Sarhan, M., Layeghy, S., and Portmann, M. (2021). Towards a standard feature set for network intrusion detection system datasets. Mobile Networks and Applications, pages 1–14.

Sharafaldin., I., Habibi Lashkari., A., and Ghorbani., A. A. (2018). Toward generating a new intrusion detection dataset and intrusion traffic characterization. In Proceedings of the 4th International Conference on Information Systems Security and Privacy - ICISSP,, pages 108–116. INSTICC, SciTePress.

Sommer, R. and Paxson, V. (2010). Outside the closed world: On using machine learning for network intrusion detection. In 2010 IEEE Symposium on Security and Privacy, pages 305–316.