Active Learning for Example Selection in Meta-Learning
Abstract
Meta-Learning has been used to select algorithms based on the features of the problems in which the algorithms can be applied. Each meta-example stores the features of a given problem and the performance information related to the candidate algorithms in the problem. The construction of a set of meta-examples may be costly, since the algorithm performance is usually defined through an empirical evaluation on the problem at hand. In this context, we proposed the use of Active Learning to select only the relevant meta-examples and hence, to reduce the need for empirical evaluations of the candidate algorithms. Experiments were performed using the kNN algorithm was as meta-learner and an uncertainty criteria was applied to select meta-examples. A significant gain in performance was yielded by selecting about 6% of the available meta-examples.
References
Brazdil, P., Soares, C., and da Costa, J. (2003). Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3):251–277.
dos Santos, P., Ludermir, T. B., and Prudêncio, R. B. C. (2004). Selection of time series forecasting models based on performance information. In 4th International Conference on Hybrid Intelligent Systems, pages 366–371.
Engels, R. and Theusinger, C. (1998). Using a data metric for preprocessing advice for data mining applications. In Prade, H., editor, Proceedings of the 13th European Conference on Artificial Intelligence (ECAI-98), pages 430–434. John Wiley & Sons.
Giraud-Carrier, C., Vilalta, R., and Brazdil, P. (2004). Introduction to the special issue on meta-learning. Machine Learning, 54(3):187–193.
Kalousis, A., Gama, J., and Hilario, M. (2004). On data and algorithms understanding inductive performance. Machine Learning, 54(3):275–312.
Kalousis, A. and Hilario, M. (2003). Representational issues in meta-learning. In Proceedings of the 20th International Conferente on Machine Learning, pages 313–320.
Kalousis, A. and Theoharis, T. (1999). Noemon: Design, implementation and performance results of an intelligent assistant for classifier selection. Intelligent Data Analysis, 3(5):319–337.
Leite, R. and Brazdil, P. (2005). Predicting relative performance of classifiers from samples. In Proceedings of the 22nd International Conference on Machine Learning.
Lindenbaum, M., Markovitch, S., and Rusakov, D. (2004). Selective sampling for nearest neighbor classifiers. Machine Learning, 54:125–152.
Makridakis, S. and Hibon, M. (2000). The M3-competition: Results, conclusions and implications. International Journal of Forecasting, 16(4):451–476.
Prudêncio, R. B. C. and Ludermir, T. B. (2004). Meta-learning approaches to selecting time series models. Neurocomputing, 61:121–137.
Prudêncio, R. B. C. and Ludermir, T. B. (2006). A machine learning approach to define weights for linear combination of forecasts. In 16th International Conference on Artificial Neural Networks, pages 274–283.
Prudêncio, R. B. C., Ludermir, T. B., and de Carvalho, F. A. T. (2004). A modal symbolic classifier to select time series models. Pattern Recognition Letters, 25(8):911–921.
Soares, C. and Brazdil, P. (2000). Zoomed ranking selection of classification algorithms based on relevant performance information. Lecture Notes in Computer Science, 1910:126–135.
Tsoumakas, G., Vrakas, D., Bassiliades, N., and Vlahavas, I. (2004). Lazy adaptive multicriteria planning. In Proceedings of the 16th European Conference on Artificial Intelligence, ECAI04, pages 693–697.
