Exploratory Landscape Analysis as Meta-Features for Hyperparameter Tuning Recommendation: A Study on SVMs
Resumo
A fundamental challenge in applying classification algorithms is selecting appropriate hyperparameter settings for a given dataset. Recently, AutoML has emerged as a paradigm for addressing this burden, with some applications based on Metalearning (MtL). By observing which algorithms and settings performed well across datasets, MtL can recommend promising configurations without extensive search. In this work, we investigate whether Exploratory Landscape Analysis (ELA) descriptors are useful as meta-features in a recommender system that predicts whether tuning Support Vector Machine (SVM) hyperparameters is worthwhile. We extracted 50 ELA features from 2D SVM optimization surfaces on 169 classification datasets. Experiments considered three meta-datasets: ELA-only, a state-of-the-art baseline (pyMFE), and their combination. These meta-datasets were evaluated using eight different algorithms and preprocessing sub-tasks. Experimental results showed that the combined meta-dataset achieved the best AUC of 0.825 when combined with the XGBoost algorithm. The Friedman-Nemenyi test confirmed it as significantly superior to the baseline. A SHAP analysis of the best model showed that 6 of the 10 most important features are ELA descriptors, confirming that landscape information provides a complementary and discriminative signal for the tune-versus-default decision.
Referências
Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281–305.
Brazdil, P., van Rijn, J. N., Soares, C., and Vanschoren, J. (2022). Metalearning: Applications to Automated Machine Learning and Data Mining. Cognitive Technologies. Springer, Cham, 2 edition. Open Access.
Cortes, C. and Vapnik, V. (1995). Support vector networks. Machine Learning, 20:273–297.
Gomes, T. A., Prudêncio, R. B., Soares, C., Rossi, A. L., and Carvalho, A. (2012). Combining meta-learning and search techniques to select parameters for support vector machines. Neurocomputing, 75(1):3–13.
Horváth, T., Mantovani, R. G., and de Carvalho, A. C. (2023). Hyper-parameter initialization of classification algorithms using dynamic time warping: A perspective on pca meta-features. Applied Soft Computing, 134:109969.
Hutter, F., Kotthoff, L., and Vanschoren, J., editors (2019). Automated Machine Learning - Methods, Systems, Challenges. Springer.
Lima Marinho, T., do Nascimento, D. C., and Pimentel, B. A. (2024). Optimization on selecting xgboost hyperparameters using meta-learning. Expert Systems, 41(9):e13611.
Lundberg, S. M., Erion, G. G., and Lee, S. (2018). Consistent individualized feature attribution for tree ensembles. CoRR, abs/1802.03888.
Malan, K. M. (2021). A survey of advances in landscape analysis for optimisation. Algorithms, 14(2).
Mantovani, R. G., Horváth, T., Rossi, A. L. D., Cerri, R., Barbon Junior, S., Vanschoren, J., and Carvalho, A. C. P. L. F. d. (2024). Better trees: an empirical study on hyperparameter tuning of classification decision tree induction algorithms. Data Min. Knowl. Discov., 38(3):1364–1416.
Mantovani, R. G., Rossi, A. L., Alcobaça, E., Vanschoren, J., and de Carvalho, A. C. (2019). A meta-learning recommender system for hyperparameter tuning: predicting when tuning improves svm classifiers. Information Sciences, 501:193–221.
Marsland, S. (2009). Machine Learning - An Algorithmic Perspective. Chapman and Hall / CRC machine learning and pattern recognition series. CRC Press.
Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., and Rudolph, G. (2011). Exploratory landscape analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, GECCO ’11, page 829–836, New York, NY, USA. Association for Computing Machinery.
Rice, J. R. (1976). The algorithm selection problem. volume 15 of Advances in Computers, pages 65–118. Elsevier.
Ridd, P. and Giraud-Carrier, C. (2014). Using metalearning to predict when parameter optimization is likely to improve classification accuracy. In Vanschoren, J., Brazdil, P., Soares, C., and Kotthoff, L., editors, Meta-learning and Algorithm Selection Workshop at ECAI 2014, pages 18–23.
Rivolli, A., Garcia, L. P., Soares, C., Vanschoren, J., and de Carvalho, A. C. (2022). Meta-features for meta-learning. Knowledge-Based Systems, 240:108101.
Vanschoren, J. (2019). Meta-Learning, pages 35–61. Springer International Publishing.
