Avaliação de Ferramentas de AutoML em Datasets de Detecção de Malwares Android
Resumo
O desenvolvimento de modelos preditivos corretos e eficazes requer um conhecimento técnico e do domínio do problema, o que muitas vezes não ocorre na prática, levando a soluções enviesadas e pouco eficazes. Ferramentas de AutoML surgiram com o propósito automatizar as etapas que envolvem o treinamento de modelos de machine learning. Neste contexto, trazemos uma avaliação de desempenho de quatro ferramentas de AutoML (Auto-Sklearn, AutoGluon, TPOT, QuickAutoML) na geração de classificadores de aplicações Android, considerando três métricas: acurácia, revocação e tempo de execução. Utilizamos sete datasets distintos para demonstrar a variação de desempenho entre as ferramentas.
Palavras-chave:
AutoML
Referências
Alaa, A. and Schaar, M. (2018). Autoprognosis: Automated clinical prognostic modeling via bayesian optimization with structured kernel learning. In International conference on machine learning, pages 139–148. PMLR.
Arp, D., Spreitzenbarth, M., Hubner, M., Gascon, H., Rieck, K., and Siemens, C. (2014). Drebin: Effective and explainable detection of android malware in your pocket. In NDSS, pages 23–26.
Bezrukavnikov, O. and Linder, R. (2021). A neophyte with AutoML: Evaluating the promises of automatic machine learning tools. https://arxiv.org/abs/2101.05840.
Cai, L., Li, Y., and Xiong, Z. (2021). Jowmdroid: Android malware detection based on feature weighting with joint optimization of weight-mapping and classifier parameters. Computers & Security, 100:102086.
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. (2020). AutoGluon-Tabular: Robust and accurate AutoML for structured data. https://arxiv.org/abs/2003.06505.
Ferreira, L., Pilastri, A., Martins, C. M., Pires, P. M., and Cortez, P. (2021). A comparison of AutoML tools for machine learning, deep learning and xgboost. In IJCNN, pages 1–8.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015). Efficient and robust automated machine learning. In NIPS, pages 2962–2970.
Guo, X., van Stein, B., and Back, T. (2019). A new approach towards the combined algorithm selection and hyper-parameter optimization problem. In IEEE SSCI, pages 2042–2049.
He, X., Zhao, K., and Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212:106622. 10.1016/j.knosys.2020.106622.
Karmaker, S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., and Veeramachaneni, K. (2021). AutoML to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 54(8):1–36.
Mahindru, A. and Sangal, A. L. (2019). DeepDroid: Feature selection approach to detect Android malware using deep learning. In IEEE 10th ICSESS, pages 16–19.
Moutaz, A. (2020). Automated malware detection in mobile app stores based on robust feature generation. Electronics, 9:435.
Nagarajah, T. and Poravi, G. (2019). A review on automated machine learning (AutoML) systems. In IEEE 5th I2CT, pages 1–6.
Olson, R. S. and Moore, J. H. (2016). TPOT: A tree-based pipeline optimization tool for automating machine learning. In Workshop on automatic machine learning, pages 66–74. PMLR.
Rahm, E. and Do, H. H. (2000). Data cleaning: Problems and current approaches. IEEE Data Eng. Bull., 23(4):3–13.
Ruth, C. (2022). Over 30 million new malware samples found in 2022 as cyber threats evolve. shorturl.at/ilNZ5.
Siqueira, G., Rodrigues, G., Kreutz, D., and Feitosa, E. (2021). QuickAutoML: Umaferramenta para treinamento automatizado de modelos de aprendizado de maquina. In VI Workshop Regional de Segurança da Informação e de Sistemas Computacionais (WRSeg). [link].
SISTO, A. (2013). AndroCrawl: studying alternative Android marketplaces. https://www.politesi.polimi.it/handle/10589/88407.
Soares, T., Siqueira, G., Barcellos, L., Sayyed, R., Vargas, L., Rodrigues, G., Assolin, J., Pontes, J., Feitosa, E., and Kreutz, D. (2021). Detecção de Malwares Android: datasets e reprodutibilidade. In VI Workshop Regional de Segurança da Informação e de Sistemas Computacionais (WRSeg).
Sun, L., Li, Z., Yan, Q., Srisa-an, W., and Pan, Y. (2016). SigPID: significant permission identification for android malware detection. In 2016 11th International Conference on Malicious and Unwanted Software (MALWARE), pages 1–8.
Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, C. B., and Farivar, R. (2019). Towards automated machine learning: Evaluation and comparison of AutoML approaches and tools. In IEEE 31st ICTAI, pages 1471–1479.
Tsamardinos, I., Charonyktakis, P., Lakiotaki, K., Borboudakis, G., Zenklusen, J. C., Juhl, H., Chatzaki, E., and Lagani, V. (2020). Just add data: Automated predictive modeling and biosignature discovery. BioRxiv.
Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., and Deng, S.-H. (2019). Hyperparameter optimization for machine learning models based on bayesian optimization. Journal of Electronic Science and Technology, 17(1):26–40.
Arp, D., Spreitzenbarth, M., Hubner, M., Gascon, H., Rieck, K., and Siemens, C. (2014). Drebin: Effective and explainable detection of android malware in your pocket. In NDSS, pages 23–26.
Bezrukavnikov, O. and Linder, R. (2021). A neophyte with AutoML: Evaluating the promises of automatic machine learning tools. https://arxiv.org/abs/2101.05840.
Cai, L., Li, Y., and Xiong, Z. (2021). Jowmdroid: Android malware detection based on feature weighting with joint optimization of weight-mapping and classifier parameters. Computers & Security, 100:102086.
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. (2020). AutoGluon-Tabular: Robust and accurate AutoML for structured data. https://arxiv.org/abs/2003.06505.
Ferreira, L., Pilastri, A., Martins, C. M., Pires, P. M., and Cortez, P. (2021). A comparison of AutoML tools for machine learning, deep learning and xgboost. In IJCNN, pages 1–8.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015). Efficient and robust automated machine learning. In NIPS, pages 2962–2970.
Guo, X., van Stein, B., and Back, T. (2019). A new approach towards the combined algorithm selection and hyper-parameter optimization problem. In IEEE SSCI, pages 2042–2049.
He, X., Zhao, K., and Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212:106622. 10.1016/j.knosys.2020.106622.
Karmaker, S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., and Veeramachaneni, K. (2021). AutoML to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 54(8):1–36.
Mahindru, A. and Sangal, A. L. (2019). DeepDroid: Feature selection approach to detect Android malware using deep learning. In IEEE 10th ICSESS, pages 16–19.
Moutaz, A. (2020). Automated malware detection in mobile app stores based on robust feature generation. Electronics, 9:435.
Nagarajah, T. and Poravi, G. (2019). A review on automated machine learning (AutoML) systems. In IEEE 5th I2CT, pages 1–6.
Olson, R. S. and Moore, J. H. (2016). TPOT: A tree-based pipeline optimization tool for automating machine learning. In Workshop on automatic machine learning, pages 66–74. PMLR.
Rahm, E. and Do, H. H. (2000). Data cleaning: Problems and current approaches. IEEE Data Eng. Bull., 23(4):3–13.
Ruth, C. (2022). Over 30 million new malware samples found in 2022 as cyber threats evolve. shorturl.at/ilNZ5.
Siqueira, G., Rodrigues, G., Kreutz, D., and Feitosa, E. (2021). QuickAutoML: Umaferramenta para treinamento automatizado de modelos de aprendizado de maquina. In VI Workshop Regional de Segurança da Informação e de Sistemas Computacionais (WRSeg). [link].
SISTO, A. (2013). AndroCrawl: studying alternative Android marketplaces. https://www.politesi.polimi.it/handle/10589/88407.
Soares, T., Siqueira, G., Barcellos, L., Sayyed, R., Vargas, L., Rodrigues, G., Assolin, J., Pontes, J., Feitosa, E., and Kreutz, D. (2021). Detecção de Malwares Android: datasets e reprodutibilidade. In VI Workshop Regional de Segurança da Informação e de Sistemas Computacionais (WRSeg).
Sun, L., Li, Z., Yan, Q., Srisa-an, W., and Pan, Y. (2016). SigPID: significant permission identification for android malware detection. In 2016 11th International Conference on Malicious and Unwanted Software (MALWARE), pages 1–8.
Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, C. B., and Farivar, R. (2019). Towards automated machine learning: Evaluation and comparison of AutoML approaches and tools. In IEEE 31st ICTAI, pages 1471–1479.
Tsamardinos, I., Charonyktakis, P., Lakiotaki, K., Borboudakis, G., Zenklusen, J. C., Juhl, H., Chatzaki, E., and Lagani, V. (2020). Just add data: Automated predictive modeling and biosignature discovery. BioRxiv.
Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., and Deng, S.-H. (2019). Hyperparameter optimization for machine learning models based on bayesian optimization. Journal of Electronic Science and Technology, 17(1):26–40.
Publicado
12/09/2022
Como Citar
SIQUEIRA, Guilherme; KREUTZ, Diego; ASSOLIN, Joner; COSTA, Estevão; MIERS, Charles; MANSILHA, Rodrigo; PONTES, Jonas; FEITOSA, Eduardo.
Avaliação de Ferramentas de AutoML em Datasets de Detecção de Malwares Android. In: SIMPÓSIO BRASILEIRO DE SEGURANÇA DA INFORMAÇÃO E DE SISTEMAS COMPUTACIONAIS (SBSEG), 22. , 2022, Santa Maria.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 302-315.
DOI: https://doi.org/10.5753/sbseg.2022.225317.
