Meta-Learning Approach for Noise Filter Algorithm Recommendation
Resumo
Preprocessing techniques can increase the quality or even enable Machine Learning algorithms. However, it is not simple to identify the preprocessing algorithms we should apply. This work proposes a methodology to recommend a noise filtering algorithm based on Meta-Learning, predicting which algorithm should be chosen based on a set of features calculated from a dataset. From synthetics datasets, we created the meta-data from an extracted set of meta-features and the f1-score performance metric calculated from the DT, KNN, and RF classifiers. To perform the suggestion, we used a meta-ranker that returns the rank of the best algorithms. We selected three noise filtering algorithms, HARF, GE, and ORBoost. To predict the f1-score, we used the PCT, RF, and KNN algorithms as meta-rankers. Our results indicate that the proposed solution acquired over 60% and 80% accuracy when considering a top-1 and top-2 approach. It also shows that the meta-rankers, when compared with a random choice and single algorithms as a baseline, provided an overall performance gain for the Machine Learning algorithm.
Palavras-chave:
meta-learning, noise detection, preprocessing, machine learning, ranking
Referências
Alcobaça, E., Siqueira, F., Rivolli, A., Garcia, L. P. F., Oliva, J. T., and de Carvalho, A. C. P. L. F. Mfe: Towards reproducible meta-feature extraction. Journal of Machine Learning Research 21 (111): 1–5, 2020.
Bilalli, B., Abelló, A., Aluja-Banet, T., and Wrembel, R. Presistant: Learning based assistant for data pre-processing. Data & Knowledge Engineering vol. 123, pp. 1–22, 2019.
Brazdil, P., Giraud-Carrier, C., Soares, C., and Vilalta, R. Metalearning - Applications to Data Mining. Cognitive Technologies. Springer, Berlin, Heidelberg, 2009.
Breiman, L. Random forests. Machine learning 45 (1): 5–32, 2001.
Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. Classification and regression trees. Routledge, New York, NY, 2017.
Cawley, G. C. and Talbot, N. L. Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers. Pattern Recognition 36 (11): 2585–2592, 2003.
Famili, A., Shen, W.-M., Weber, R., and Simoudis, E. Data preprocessing and intelligent data analysis. Intelligent data analysis 1 (1): 3–23, 1997.
Fayyad, U. M., Haussler, D., and Stolorz, P. E. Kdd for science data analysis: Issues and examples. In Second International Conference on Knowledge Discovery & Data Mining (KDD). AAAI Press, Portland, OR, pp. 50–56, 1996.
Frénay, B. and Verleysen, M. Classification in the presence of label noise: A survey. IEEE Transactions on Neural Networks and Learning Systems 25 (5): 845–869, 2014.
Freund, Y. and Schapire, R. E. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 (1): 119–139, 1995.
Garcia, L. P., de Carvalho, A. C., and Lorena, A. C. Noise detection in the meta-learning level. Neurocomputing vol. 176, pp. 14–25, 2016.
García, S., Luengo, J., and Herrera, F. Data preprocessing in data mining. Vol. 72. Springer, Cham, Switzerland, 2015.
Gupta, S. and Gupta, A. Dealing with noise problem in machine learning data-sets: A systematic review. Procedia Computer Science vol. 161, pp. 466–474, 2019.
Hutter, F., Kotthoff, L., and Vanschoren, J. Automated machine learning: methods, systems, challenges. Springer Nature, Cham, Switzerland, 2019.
Karmaker, A. and Kwek, S. A boosting approach to remove class label noise. International Journal of Hybrid Intelligent Systems 3 (3): 169–177, 2006.
Koplowitz, J. and Brown, T. A. On the relation of performance to editing in nearest neighbor rules. Pattern Recognition 13 (3): 251–255, 1981.
Mitchell, T. M. Machine Learning. McGraw Hill series in computer science. McGraw Hill, New York, NY, 1997.
Morales, P., Luengo, J., Garcia, L. P., Lorena, A. C., de Carvalho, A. C., and Herrera, F. The noisefiltersr package: Label noise preprocessing in r. The R Journal 9 (1): 219–228, 2017.
Munson, M. A. A study on the importance of and time spent on different modeling steps. ACM SIGKDD Explorations Newsletter 13 (2): 65–71, 2012.
Nagarajah, T. and Poravi, G. A review on automated machine learning (automl) systems. In 5th International Conference for Convergence in Technology (I2CT). IEEE, Bombay, India, pp. 1–6, 2019.
Parmezan, A. R. S., Lee, H. D., Spolaôr, N., and Wu, F. C. Automatic recommendation of feature selection algorithms based on dataset characteristics. Expert Systems with Applications vol. 185, pp. 115589, 2021.
Rice, J. R. The algorithm selection problem. Advances in Computers vol. 15, pp. 65–118, 1976.
Rivolli, A., Garcia, L. P., Soares, C., Vanschoren, J., and de Carvalho, A. C. Meta-features for meta-learning. Knowledge-Based Systems vol. 240, pp. 108101, 2022.
Russell, S. J. and Norvig, P. Artificial Intelligence: a modern approach. Pearson, Prentice Hall Upper Saddle River, NJ, USA, 2009.
Sluban, B., Gamberger, D., and Lavrač, N. Ensemble-based noise detection: noise ranking and visual performance evaluation. Data Mining and Knowledge Discovery 28 (2): 265–303, 2014.
Smith-Miles, K. A. Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Computing Surveys 41 (1): 1–25, 2008.
Todorovski, L., Blockeel, H., and Dzeroski, S. Ranking with predictive clustering trees. In European Conference on Machine Learning (ECML). Springer, Berlin, Heidelberg, pp. 444–455, 2002.
Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, C. B., and Farivar, R. Towards automated machine learning: Evaluation and comparison of automl approaches and tools. In 31st International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, Portland, OR, pp. 1471–1479, 2019.
Vanschoren, J. Meta-learning. In Automated Machine Learning. Springer Nature, Cham, Switzerland, pp. 35–61, 2019.
Wilson, D. L. Asymptotic properties of nearest neighbor rules using edited data. IEEE Transactions on Systems, Man, and Cybernetics SMC-2 (3): 408–421, 1972.
Wirth, R. and Hipp, J. Crisp-dm: Towards a standard process model for data mining. In 4th International Conference on the Practical Application of Knowledge Discovery and Data Mining. AAAI Press, New York, NY, pp. 29–39, 2000.
Zar, J. H. Spearman rank correlation: overview. Wiley StatsRef: Statistics Reference Online, 2014.
Zhu, X. and Wu, X. Class noise vs. attribute noise: A quantitative study. Artificial Intelligence Review 22 (3): 177–210, 2004.
Bilalli, B., Abelló, A., Aluja-Banet, T., and Wrembel, R. Presistant: Learning based assistant for data pre-processing. Data & Knowledge Engineering vol. 123, pp. 1–22, 2019.
Brazdil, P., Giraud-Carrier, C., Soares, C., and Vilalta, R. Metalearning - Applications to Data Mining. Cognitive Technologies. Springer, Berlin, Heidelberg, 2009.
Breiman, L. Random forests. Machine learning 45 (1): 5–32, 2001.
Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. Classification and regression trees. Routledge, New York, NY, 2017.
Cawley, G. C. and Talbot, N. L. Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers. Pattern Recognition 36 (11): 2585–2592, 2003.
Famili, A., Shen, W.-M., Weber, R., and Simoudis, E. Data preprocessing and intelligent data analysis. Intelligent data analysis 1 (1): 3–23, 1997.
Fayyad, U. M., Haussler, D., and Stolorz, P. E. Kdd for science data analysis: Issues and examples. In Second International Conference on Knowledge Discovery & Data Mining (KDD). AAAI Press, Portland, OR, pp. 50–56, 1996.
Frénay, B. and Verleysen, M. Classification in the presence of label noise: A survey. IEEE Transactions on Neural Networks and Learning Systems 25 (5): 845–869, 2014.
Freund, Y. and Schapire, R. E. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 (1): 119–139, 1995.
Garcia, L. P., de Carvalho, A. C., and Lorena, A. C. Noise detection in the meta-learning level. Neurocomputing vol. 176, pp. 14–25, 2016.
García, S., Luengo, J., and Herrera, F. Data preprocessing in data mining. Vol. 72. Springer, Cham, Switzerland, 2015.
Gupta, S. and Gupta, A. Dealing with noise problem in machine learning data-sets: A systematic review. Procedia Computer Science vol. 161, pp. 466–474, 2019.
Hutter, F., Kotthoff, L., and Vanschoren, J. Automated machine learning: methods, systems, challenges. Springer Nature, Cham, Switzerland, 2019.
Karmaker, A. and Kwek, S. A boosting approach to remove class label noise. International Journal of Hybrid Intelligent Systems 3 (3): 169–177, 2006.
Koplowitz, J. and Brown, T. A. On the relation of performance to editing in nearest neighbor rules. Pattern Recognition 13 (3): 251–255, 1981.
Mitchell, T. M. Machine Learning. McGraw Hill series in computer science. McGraw Hill, New York, NY, 1997.
Morales, P., Luengo, J., Garcia, L. P., Lorena, A. C., de Carvalho, A. C., and Herrera, F. The noisefiltersr package: Label noise preprocessing in r. The R Journal 9 (1): 219–228, 2017.
Munson, M. A. A study on the importance of and time spent on different modeling steps. ACM SIGKDD Explorations Newsletter 13 (2): 65–71, 2012.
Nagarajah, T. and Poravi, G. A review on automated machine learning (automl) systems. In 5th International Conference for Convergence in Technology (I2CT). IEEE, Bombay, India, pp. 1–6, 2019.
Parmezan, A. R. S., Lee, H. D., Spolaôr, N., and Wu, F. C. Automatic recommendation of feature selection algorithms based on dataset characteristics. Expert Systems with Applications vol. 185, pp. 115589, 2021.
Rice, J. R. The algorithm selection problem. Advances in Computers vol. 15, pp. 65–118, 1976.
Rivolli, A., Garcia, L. P., Soares, C., Vanschoren, J., and de Carvalho, A. C. Meta-features for meta-learning. Knowledge-Based Systems vol. 240, pp. 108101, 2022.
Russell, S. J. and Norvig, P. Artificial Intelligence: a modern approach. Pearson, Prentice Hall Upper Saddle River, NJ, USA, 2009.
Sluban, B., Gamberger, D., and Lavrač, N. Ensemble-based noise detection: noise ranking and visual performance evaluation. Data Mining and Knowledge Discovery 28 (2): 265–303, 2014.
Smith-Miles, K. A. Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Computing Surveys 41 (1): 1–25, 2008.
Todorovski, L., Blockeel, H., and Dzeroski, S. Ranking with predictive clustering trees. In European Conference on Machine Learning (ECML). Springer, Berlin, Heidelberg, pp. 444–455, 2002.
Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, C. B., and Farivar, R. Towards automated machine learning: Evaluation and comparison of automl approaches and tools. In 31st International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, Portland, OR, pp. 1471–1479, 2019.
Vanschoren, J. Meta-learning. In Automated Machine Learning. Springer Nature, Cham, Switzerland, pp. 35–61, 2019.
Wilson, D. L. Asymptotic properties of nearest neighbor rules using edited data. IEEE Transactions on Systems, Man, and Cybernetics SMC-2 (3): 408–421, 1972.
Wirth, R. and Hipp, J. Crisp-dm: Towards a standard process model for data mining. In 4th International Conference on the Practical Application of Knowledge Discovery and Data Mining. AAAI Press, New York, NY, pp. 29–39, 2000.
Zar, J. H. Spearman rank correlation: overview. Wiley StatsRef: Statistics Reference Online, 2014.
Zhu, X. and Wu, X. Class noise vs. attribute noise: A quantitative study. Artificial Intelligence Review 22 (3): 177–210, 2004.
Publicado
28/11/2022
Como Citar
B. PIO, P.; GARCIA, L. P. F.; RIVOLLI, A..
Meta-Learning Approach for Noise Filter Algorithm Recommendation. In: SYMPOSIUM ON KNOWLEDGE DISCOVERY, MINING AND LEARNING (KDMILE), 10. , 2022, Campinas/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 186-193.
ISSN 2763-8944.
DOI: https://doi.org/10.5753/kdmile.2022.227958.
