Noise filter with hyperparameter recommendation: a meta-learning approach
Resumo
Applying Machine Learning (ML) algorithms to a dataset can be time-consuming. It usually involves, not only selecting and fine-tuning the algorithm, but also other steps, such as data preprocessing. To reduce this time, the whole or a subset of this process has been automated by Automated ML (AutoML) techniques, which can include Bayesian Optimization, Genetic Programming, and Meta-Learning techniques. However, despite it often being a necessary stage, preprocessing is commonly not well handled in AutoML tools. In this work, we propose and experimentally investigate the use of meta-learning to recommend noise detection algorithms and the values for their hyperparameters. The proposed approach produces a ranking of the best noise filters for a given dataset, reducing the development cost of ML-based solutions and improving their predictive performance. To validate the process, we generated 10740 noisy datasets, which we describe using 97 meta-features. For each dataset, we applied 8 noise filters, which increased to 27 when we added variations of hyperparameter values. Next, we applied 4 ML algorithms to this data and created a performance ranking, which we used as a meta-target to induce 3 meta-regressors. We compared these 3 meta-regressors and the results with and without hyperparameters for the noise filters. According to the experimental results, the introduction of hyperparameter recommendation resulted in a higher gain in the F1-Score performance metric. However, it came at the cost of lower accuracy in the Top-K ranking evaluation.
Referências
Brazdil, P., van Rijn, J. N., Soares, C., and Vanschoren, J. (2022). Metalearning: Applications to Automated Machine Learning and Data Mining. Springer Nature.
de Morais, R. F., Miranda, P. B., and Silva, R. M. (2016). A meta-learning method to select under-sampling algorithms for imbalanced data sets. In 5th Brazilian Conference on Intelligent Systems, pages 385–390. IEEE.
Demsǎr, J. (2006). Statistical comparisons of classifiers over multiple datasets. The Journal of Machine learning research, 7:1–30.
Famili, A., Shen, W.-M., Weber, R., and Simoudis, E. (1997). Data preprocessing and intelligent data analysis. Intelligent Data Analysis, 1(1):3–23.
Fayyad, U. M., Haussler, D., and Stolorz, P. E. (1996). Kdd for science data analysis: Issues and examples. In Second International Conference on Knowledge Discovery & Data Mining, pages 50–56, Portland, OR. AAAI Press.
Frénay, B. and Verleysen, M. (2013). Classification in the presence of label noise: a survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5):845–869.
Garcia, L. P., de Carvalho, A. C., and Lorena, A. C. (2016a). Noise detection in the meta-learning level. Neurocomputing, 176:14–25.
Garcia, L. P., Lorena, A. C., Matwin, S., and de Carvalho, A. C. (2016b). Ensembles of label noise filters: a ranking approach. Data Mining and Knowledge Discovery, 30(5):1192–1216.
Garcia, L. P. F., Lorena, A. C., and Carvalho, A. C. (2012). A study on class noise detection and elimination. In Brazilian Symposium on Neural Networks (BRACIS), pages 13–18.
García, S., Luengo, J., and Herrera, F. (2015). Data preprocessing in data mining, volume 72. Springer, Cham, Switzerland, 1 edition.
Gupta, S. and Gupta, A. (2019). Dealing with noise problem in machine learning data-sets: A systematic review. Procedia Computer Science, 161:466–474.
Karmaker, A. and Kwek, S. (2006). A boosting approach to remove class label noise. International Journal of Hybrid Intelligent Systems, 3(3):169–177.
Koplowitz, J. and Brown, T. A. (1981). On the relation of performance to editing in nearest neighbor rules. Pattern Recognition, 13(3):251–255.
Leyva, E., González, A., and Pérez, R. (2013). Knowledge-based instance selection: A compromise between efficiency and versatility. Knowledge-Based Systems, 47:65–76.
Martínez-Plumed, F., Contreras-Ochando, L., Ferri, C., Hernández-Orallo, J., Kull, M., Lachiche, N., Ramirez-Quintana, M. J., and Flach, P. (2019). Crisp-dm twenty years later: From data mining processes to data science trajectories. IEEE Transactions on Knowledge and Data Engineering, 33(8):3048–3061.
Miranda, A. L., Garcia, L. P. F., Carvalho, A. C., and Lorena, A. C. (2009). Use of classification algorithms in noise detection and elimination. In International Conference on Hybrid Artificial Intelligence Systems, pages 417–424.
Nagarajah, T. and Poravi, G. (2019). A review on automated machine learning (automl) systems. In 5th International Conference for Convergence in Technology, pages 1–6, Bombay, India. IEEE.
Nguyen,G.,Dlugolinsky,S.,Bobák,M.,Tran,V.,LópezGarcía,A ́.,Heredia,I.,Malík,P., and Hluchý, L. (2019). Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey. Artificial Intelligence Review, 52(1):77–124.
Parmezan, A. R. S., Lee, H. D., Spolaôr, N., and Wu, F. C. (2021). Automatic recommendation of feature selection algorithms based on dataset characteristics. Expert Systems with Applications, 185:115589.
Pio, P. B., Garcia, L. P., and Rivolli, A. (2022). Meta-learning approach for noise filter algorithm recommendation. In X Symposium on Knowledge Discovery, Mining and Learning, volume InPress.
Rice, J. R. (1976). The algorithm selection problem. Advances in Computers, 15:65–118.
Rivolli, A., Garcia, L. P., Soares, C., Vanschoren, J., and de Carvalho, A. C. (2022). Meta-features for meta-learning. Knowledge-Based Systems, 240:108101.
Sluban, B., Gamberger, D., and Lavracˇ, N. (2014). Ensemble-based noise detection: noise ranking and visual performance evaluation. Data Mining and Knowledge Discovery, 28(2):265–303.
Smith, M. R. and Martinez, T. (2011). Improving classification accuracy by identifying and removing instances that should be misclassified. In International Joint Conference on Neural Networks, pages 2690–2697.
Smith-Miles, K. (2008). Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Computing Surveys, 41(1):1–25.
Tomek, I. (1976). An experiment with the edited nearest-neighbor rule. IEEE Transactions on Systems, Man, and Cybernetics, SMC-6(6):448–452.
Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, C. B., and Farivar, R. (2019). Towards automated machine learning: Evaluation and comparison of automl approaches and tools. In 31st International Conference on Tools with Artificial Intelligence, pages 1471–1479, Portland, OR. IEEE.
Vanschoren, J. (2019). Meta-learning. In Automated Machine Learning, pages 35–61. Springer Nature, Cham, Switzerland.
Wheway, V. (2001). Using boosting to detect noisy data. In Pacific Rim International Conference on Artificial Intelligence, pages 123–130.
Zhu, X. and Wu, X. (2004). Class noise vs. attribute noise: A quantitative study. Artificial Intelligence Review, 22(3):177–210.
