Surrogate Models for Pipeline Recommendation in Multi-Label Classification
Resumo
Multi-Label Classification (MLC) is a challenging task where an instance is simultaneously classified into two or more classes. Due to its possible high dimensionality, feature selection is a crucial step in the classification pipeline. While AutoML can be used to optimize this pipeline selection, its empirical evaluation incurs prohibitive computational costs. This paper proposes a surrogate-based approach for efficient pipeline recommendation in MLC, comprising both classification and feature-selection algorithms. For this, a Multi-Output regressor model, trained on optimization histories, simultaneously predicts F1-score and model size of candidate pipelines. Coupled with an optimization algorithm, this method explores the search space without executing configurations in real time. Results prove the surrogate’s high predictive accuracy (R2 > 0.90 for performance and > 0.98 for structural cost) and its practical value. By applying this prior knowledge, the multi-objective approach identifies competitive configurations that greatly reduce feature selection time (e.g., from 280s to 0.15s) while recommending pipelines with a memory footprint orders of magnitude smaller (under 1 MB). Ultimately, this optimal trade-off makes AutoML viable in scenarios with severe temporal and storage restrictions.
Referências
Bhosekar, A. and Ierapetritou, M. (2018). Advances in surrogate based modeling, feasibility analysis, and optimization: A review. Computers & Chemical Engineering, 108:250–267.
Chen, T. and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA. ACM.
de Sá, A. G. C., Freitas, A. A., and Pappa, G. L. (2018). Automated selection and configuration of multi-label classification algorithms with grammar-based genetic programming. In Parallel Problem Solving from Nature, volume 11101, pages 308–320.
Del Valle, A. M. (2025). Aprendizado de máquina automatizado multiobjetivo para classificação multirrótulo. Tese (doutorado em ciência da computação), Universidade Federal de São Carlos, São Carlos.
Del Valle, A. M., Mantovani, R. G., and Cerri, R. (2023). AutoMMLC: An Automated and Multi-objective Method for Multi-label Classification. In Intelligent Systems, pages 291–306.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015). Efficient and Robust Automated Machine Learning. In Advances in Neural Information Processing Systems, volume 28.
He, X., Zhao, K., and Chu, X. (2021). AutoML: A survey of the state-of-the-art. Knowledge-Based Systems, 212:106622.
Hutter, F., Hoos, H. H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization, volume 6683, pages 507–523.
Laadan, D., Vainshtein, R., Curiel, Y., Katz, G., and Rokach, L. (2019). RankML: a Meta Learning-Based Approach for Pre-Ranking Machine Learning Pipelines.
Moyano, J. M., Gibaja, E. L., Cios, K. J., and Ventura, S. (2020). Combining multi-label classifiers based on projections of the output space using evolutionary algorithms. Knowledge-Based Systems, 196:105770.
Nguyen, T.-D., Maszczyk, T., Musial, K., Zöller, M.-A., and Gabrys, B. (2020). Avatar - machine learning pipeline evaluation using surrogate model. In International Symposium on Intelligent Data Analysis, page 352–365.
Palmes, P. P., Kishimoto, A., Marinescu, R., Ram, P., and Daly, E. (2023). Designing machine learning pipeline toolkit for automl surrogate modeling optimization. Proceedings of the JuliaCon Conferences, 1(1):129.
Sechidis, K., Tsoumakas, G., and Vlahavas, I. (2011). On the Stratification of Multi-label Data. In Machine Learning and Knowledge Discovery in Databases, pages 145–158.
Theng, D. and Bhoyar, K. K. (2024). Feature selection techniques for machine learning: a survey of more than two decades of research. Knowledge & Information Systems, 66(3):1575–1637.
Tsoumakas, G. and Katakis, I. (2007). Multi-label classification: An overview. International Journal of Data Warehousing and Mining, 3(3):1–13.
Wever, M., Mohr, F., and Hüllermeier, E. (2019). Automating multi-label classification extending ml-plan. In ICML 2019 Workshop on Automated Machine Learning.
Wever, M., Tornede, A., Mohr, F., and Hüllermeier, E. (2021). AutoML for multi-label classification: Overview and empirical evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3037–3054.
Zhang, M.-L. and Zhou, Z.-H. (2014). A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819–1837.
