A Monitoring and Threat Detection System Using Stream Processing as a Virtual Function for Big Data
Resumo
The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. As a consequence, fast real-time threat detection is mandatory for security guarantees. In addition, Network Function Virtualization (NFV) provides new opportunities for efficient and low-cost security solutions. We propose a fast and efficient threat detection system based on stream processing and machine learning algorithms. The main contributions of this work are i) a novel monitoring threat detection system based on stream processing; ii) two datasets, first a dataset of synthetic security data containing both legitimate and malicious traffic, and the second, a week of real traffic of a telecommunications operator in Rio de Janeiro, Brazil; iii) a data pre-processing algorithm, a normalizing algorithm and an algorithm for fast feature selection based on the correlation between variables; iv) a virtualized network function in an open-source platform for providing a real-time threat detection service; v) near-optimal placement of sensors through a proposed heuristic for strategically positioning sensors in the network infrastructure, with a minimum number of sensors; and, finally, vi) a greedy algorithm that allocates on demand a sequence of virtual network functions.
Referências
Andreoni Lopez, M., Mattos, D. e Duarte, O. (2017a). Evaluating allocation heuristics for an efficient virtualNetwork Function chaining. Em 2016 7th International Conference on the Network of the Future, NOF 2016.
Andreoni Lopez, M., Mattos, D. M. F. e Duarte, O. C. M. B. (2016). An elastic intrusion detection system forsoftware networks. Annales des Telecommunications/Annals of Telecommunications, 71(11-12):595-605.
Andreoni Lopez, M., Mattos, D. M. F., Duarte, O. C. M. B. e Pujolle, G. (2018b). A fast unsupervised preprocess-ing method for network monitoring. Annals of Telecommunications.
Andreoni Lopez, M., Sanz, I. J., Ferrazani Mattos, D. M., Duarte, O. C. M. B. e Pujolle, G. (2017b). CATRACA:uma Ferramenta para Classificagéo e Andlise Trafego Escalavel Baseada em Processamento por Fluxo. EmSaldo de Ferramentas do XVII SBSeg’2017, paginas 788-795.
Andreoni Lopez, M., Silva, S. R., Alvarenga, I. D., Rebello, G. A. F, Sanz, J. I., Lobato, A. G. P., Mattos, D.M. F,, Duarte, O. C. M. B. e Pujolle, G. (2017c). Collecting and Characterizing a Real Broadband AccessNetwork Traffic Dataset. Em /JEEE/IFIP Ist Cyber Security in Networking Conference (CSNet’17), Rio deJaneiro, Brazil.
Apache Software Foundation (2017). Apache Metron. https://cwiki.apache.org/confluence/display/METRON/About+Metron. Acessado em 04/10/2017.
Carbone, P., Fora, G., Ewen, S., Haridi, S. e Tzoumas, K. (2015). Lightweight Asynchronous Snapshots forDistributed Dataflows. Computing Research Repository (CoRR), abs/1506.0.
Du, Y., Liu, J., Liu, Fe Chen, L. (2014). A real-time anomalies detection system based on streaming technol-ogy. Em Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC),volume 2, paginas 275-279. IEEE.
Franklin, M. (2013). The Berkeley Data Analytics Stack: Present and future. Em /EEE International Conferenceon Big Data, paginas 2—3. IEEE.
He, G., Tan, C., Yu, D. e Wu, X. (2015). A real-time network traffic anomaly detection system based on storm.Em Proceedings - 2015 7th International Conference on Intelligent Human-Machine Systems and Cybernetics,THMSC 2015, volume 1, paginas 153-156.
Hu, P., Li, H., Fu, H., Cansever, D. e Mohapatra, P. (2015). Dynamic defense strategy against advanced persistentthreat with insiders. Em JEEE Conference on Computer Communications (INFOCOM), paginas 747-755.
Jirsik, T., Cermak, M., Tovarnak, D. e Celeda, P. (2017). Toward Stream-Based IP Flow Analysis. JEEE Commu-nications Magazine, 55(7):70—76.
Mayhew, M., Atighetchi, M., Adler, A. e Greenstadt, R. (2015). Use of machine learning in big data analytics forinsider threat detection. Em JEEE MILCOM, paginas 915-922.
Mylavarapu, G., Thomas, J. e TK, A. K. (2015). Real-Time Hybrid Intrusion Detection System Using ApacheStorm. Em /7th International Conference on High Performance Computing and Communications, paginas1436-1441. IEEE.
Santos, O. (2015). Big data analytics and netflow. Em Network Security with NetFlow and IPFIX: Big DataAnalytics for Information Security. Acessado em 29/08/2017.
Sanz, I. J., Alvarenga, I. D., Andreoni Lopez, M., Mauricio, L. A. F., Mattos, D. M. F., Rubinstein, M. G. e Duarte,O. C. M. B. (2017). Uma avaliagéo de desempenho de seguranga definida por software através de cadeias defungdes de rede. Em XVII SBSeg 2017.
Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J. M., Kulkarni, S., Jackson, J., Gade, K., Fu, M.,Donham, J., Bhagat, N., Mittal, S. e Ryaboy, D. (2014). Storm@Twitter. Em ACM SIGMOD InternationalConference on Management of Data, paginas 147-156. ACM.
Wood, T., Shenoy, P., Venkataramani, A. e Yousif, M. (2009). Sandpiper: Black-box and gray-box resourcemanagement for virtual machines. Computer Networks, 53(17):2923-2938.
Wu, K., Zhang, K., Fan, W., Edwards, A. e Yu, P. S. (2014). RS-Forest: A Rapid Density Estimator for StreamingAnomaly Detection. Em JEEE International Conference on Data Mining (ICDM), paginas 600-609.
Zhao, S., Chandrashekar, M., Lee, Y. e Medhi, D. (2015). Real-time network anomaly detection system usingmachine learning. Em //th International Conference on the Design of Reliable Communication Networks(DRCN), paginas 267-270. IEEE.
