Evaluation of the Stacking Algorithm on Biomedical Data
Abstract
Stacking is a well studied ensemble technique, but some of its aspects still need to be explored, e.g., there are no recommendations on which and how many algorithms should be used at level-0 or even which algorithm should be used to compose the level-1 meta-classifier. The literature indicates the meta-algorithm at level-1 should be simple, and Naive Bayes has been typically used in studies. This study analyzed stacking in medical datasets, using three different paradigms of machine learning algorithms to compose the meta-classifier. The experiments indicate simple meta-algorithms do not provide good results, and therefore, the meta-classifier must have a degree of complexity for it to achieve a good performance.
References
Aha, D. W., Kibler, D. & Albert, M. K. (1991) “Instance based learning algorithms” In Machine Learning, p. 37–66.
Benjamini, Y. & Hochberg, Y. (1995) “Controlling the false discovery rate: a practical and powerful approach to multiple testing”, In Journal of the Royal Statistical Society Series B, v. 57, p. 289–300.
Bradley, A. P. (1997) “The use of the area under the ROC curve in the evaluation of machine learning algorithms”. Pattern Recognition 30(7), 1145–1159.
Chickering, D. M., Heckerman, D. & Meek, C. (2005) “Learning of Bayesian Networks is NP – Hard” In Journal of Machine Learning Research, 5, p 1287–1330.
Cohen, W. W. (1995) “Fast effective rule induction” In Proceedings of Twelfth International Conference on Machine Learning, p. 115–123.
Dzeroski, S. & Zenko B. (2002) “Is combining Classifiers Better than Selecting the Best One?” In Proceedings of the 19th International Conference on Machine Learning, Morgan Kaufmann Publishers, San Francisco.
Frank, A. & Asuncion, A. (2010) “UCI Machine Learning Repository”, [link], School of Information and Computer Science, University of California at Irvine, Irvine CA.
Friedman, M. (1940) “A comparison of alternative tests of significance for the problem of m rankings”. The Annals of Mathematical Statistics 11(1), 86–92.
Haykin, S. (1998) Neural networks: a comprehensive foundation, 2nd edition, Pearson Education, London.
Iba, W. & Langley, P. (1992) “Induction of One – Level Decision Trees” In Proceedings of the Ninth International Conference on Machine Learning.
Nemenyi, P. B. (1963) Distribution-free multiple comparisons, PhD. Thesis, Princeton University.
Pereira, M. & Schmitz, A. (2010) "Inteligência Artificial e Geotecnologias Emergentes Aplicadas em Estudos Ecoepidemiológicos de Malária no Município de Bragança-Pará, Brasil, no Período de 2006 a 2008", In Proceedings do X Workshop de Informática Médica, Congresso da Sociedade Brasileira de Computação, p. 1630–1640, Belo Horizonte.
Pollettini, J. T., Tinos, R., Panico, S., Daneluzzi, J. C. & Macedo, A. A. (2009) "Vigilância em atenção básica à saúde a partir do uso de relevance feedback para classificação de pacientes em diferentes níveis de cuidado em saúde", In Proceedings do IX Workshop de Informática Médica, Congresso da Sociedade Brasileira de Computação, p. 1945–1954, Bento Gonçalves.
Quinlan, J. R. (1993) C4.5: programs for machine learning, Morgan Kaufmann, San Francisco.
Rish, I. (2001) “An empirical study of the naive Bayes classifier”, In IJCAI Workshop on Empirical Methods in Artificial Intelligence, p. 41–46.
Seewald, A. K. (2002) “How to make Stacking Better and Faster While Also Taking Care of an Unknown Weakness”, In Proceedings of the 19th International Conference on Machine Learning, p. 554–561, Morgan Kaufmann Publishers, Sydney.
Seewald, A. K. (2002) “Exploring the Parameter State Space of Stacking”, In Proceedings of the 2002 IEEE International Conference of Data Mining (ICDM'02), p. 685–688.
Tanwani, A. K., Afridi, J. Shafiq, M. Z. & Farroq, M. (2009) “Guidelines to Select Machine Learning Scheme for Classification of Biomedical Datasets”, C.Pizzuti, M.D. Ritchie, & M. Giacobini, LNCS 5483, Springer-Verlag Berlin Heidelberg 2009, p. 128–139.
Vapnik, V. N. (1998) Statistical learning theory, Wiley Interscience, USA.
Witten, I. H. & Frank, E. (2005) Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, 2. ed.: Morgan Kaufmann.
Wolpert, D. H. (1992) Stacked Generalization. In Neural Networks, p. 241–260.
Benjamini, Y. & Hochberg, Y. (1995) “Controlling the false discovery rate: a practical and powerful approach to multiple testing”, In Journal of the Royal Statistical Society Series B, v. 57, p. 289–300.
Bradley, A. P. (1997) “The use of the area under the ROC curve in the evaluation of machine learning algorithms”. Pattern Recognition 30(7), 1145–1159.
Chickering, D. M., Heckerman, D. & Meek, C. (2005) “Learning of Bayesian Networks is NP – Hard” In Journal of Machine Learning Research, 5, p 1287–1330.
Cohen, W. W. (1995) “Fast effective rule induction” In Proceedings of Twelfth International Conference on Machine Learning, p. 115–123.
Dzeroski, S. & Zenko B. (2002) “Is combining Classifiers Better than Selecting the Best One?” In Proceedings of the 19th International Conference on Machine Learning, Morgan Kaufmann Publishers, San Francisco.
Frank, A. & Asuncion, A. (2010) “UCI Machine Learning Repository”, [link], School of Information and Computer Science, University of California at Irvine, Irvine CA.
Friedman, M. (1940) “A comparison of alternative tests of significance for the problem of m rankings”. The Annals of Mathematical Statistics 11(1), 86–92.
Haykin, S. (1998) Neural networks: a comprehensive foundation, 2nd edition, Pearson Education, London.
Iba, W. & Langley, P. (1992) “Induction of One – Level Decision Trees” In Proceedings of the Ninth International Conference on Machine Learning.
Nemenyi, P. B. (1963) Distribution-free multiple comparisons, PhD. Thesis, Princeton University.
Pereira, M. & Schmitz, A. (2010) "Inteligência Artificial e Geotecnologias Emergentes Aplicadas em Estudos Ecoepidemiológicos de Malária no Município de Bragança-Pará, Brasil, no Período de 2006 a 2008", In Proceedings do X Workshop de Informática Médica, Congresso da Sociedade Brasileira de Computação, p. 1630–1640, Belo Horizonte.
Pollettini, J. T., Tinos, R., Panico, S., Daneluzzi, J. C. & Macedo, A. A. (2009) "Vigilância em atenção básica à saúde a partir do uso de relevance feedback para classificação de pacientes em diferentes níveis de cuidado em saúde", In Proceedings do IX Workshop de Informática Médica, Congresso da Sociedade Brasileira de Computação, p. 1945–1954, Bento Gonçalves.
Quinlan, J. R. (1993) C4.5: programs for machine learning, Morgan Kaufmann, San Francisco.
Rish, I. (2001) “An empirical study of the naive Bayes classifier”, In IJCAI Workshop on Empirical Methods in Artificial Intelligence, p. 41–46.
Seewald, A. K. (2002) “How to make Stacking Better and Faster While Also Taking Care of an Unknown Weakness”, In Proceedings of the 19th International Conference on Machine Learning, p. 554–561, Morgan Kaufmann Publishers, Sydney.
Seewald, A. K. (2002) “Exploring the Parameter State Space of Stacking”, In Proceedings of the 2002 IEEE International Conference of Data Mining (ICDM'02), p. 685–688.
Tanwani, A. K., Afridi, J. Shafiq, M. Z. & Farroq, M. (2009) “Guidelines to Select Machine Learning Scheme for Classification of Biomedical Datasets”, C.Pizzuti, M.D. Ritchie, & M. Giacobini, LNCS 5483, Springer-Verlag Berlin Heidelberg 2009, p. 128–139.
Vapnik, V. N. (1998) Statistical learning theory, Wiley Interscience, USA.
Witten, I. H. & Frank, E. (2005) Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, 2. ed.: Morgan Kaufmann.
Wolpert, D. H. (1992) Stacked Generalization. In Neural Networks, p. 241–260.
Published
2011-07-19
How to Cite
CAFFÉ, Maria Izabela R.; PEREZ, Pedro Santoro; BARANAUSKAS, José Augusto.
Evaluation of the Stacking Algorithm on Biomedical Data. In: BRAZILIAN SYMPOSIUM ON COMPUTING APPLIED TO HEALTH (SBCAS), 11. , 2011, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2011
.
p. 1830-1839.
ISSN 2763-8952.
