Improving Multilabel Text Classification with Stacking and Recurrent Neural Networks

  • Rodrigo Mansueli UEM
  • Marcos Aurélio Domingues UEM
  • Valéria Delisandra Feltrim UEM


Multilabel text classification can be defined as a mapping function that categorizes a text in natural language into one or more labels defined by the scope of a problem. In this work we propose an architecture of stacked classifiers for multilabel text classification. The proposed models use an LSTM recurrent neural network in the first stage of the stack and different multilabel classifiers in the second stage. We evaluated our proposal in two datasets well-known in the literature (TMDB and EUR-LEX Subject Matters), and the results showed that the proposed stack consistently outperforms the baselines.
Palavras-chave: Machine Learning, Multilabel Classification, Stacking, Recurrent Neural Network


Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. 1992. A Training Algorithm for Optimal Margin Classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory (Pittsburgh, Pennsylvania, USA) (COLT ’92). Association for Computing Machinery, New York, NY, USA, 144–152.

Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1724–1734.

Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks. Machine Learning 20, 3 (1995), 273–297.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.

Katti Faceli, Ana Carolina Lorena, João Gama, and André C. P. L. F de Carvalho. 2019. Inteligência Artificial - Uma Abordagem de Aprendizado de Máquina (3 ed.).Grupo Gen - LTC.

Francisco Herrera, Francisco Charte, Antonio J. Rivera, and María J. del Jesus. 2016. Multilabel Classification. Springer International Publishing.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780.

Anwesha Law and Ashish Ghosh. 2019. Multi-label classification using a cascade of stacked autoencoder and extreme learning machines. Neurocomputing 358 (2019), 222–234.

Rémi Lebret and Ronan Collobert. 2014. Word Embeddings through Hellinger PCA. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics,Gothenburg, Sweden, 482–490.

Eneldo Loza Mencía and Johannes Fürnkranz. 2010. Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain. In Semantic Processing of Legal Texts: Where the Language of Law Meets the Law of Language. Springer Berlin Heidelberg, Berlin, Heidelberg, 192–215.

Eneldo Loza Mencía and Frederik Janssen. 2016. Learning rules for multi-label classification: a stacking and a separate-and-conquer approach. Machine Learning 105, 1 (2016), 77–126.

Rafael B. Mangolin, Rodolfo M. Pereira, Alceu S. Britto, Carlos N. Silla, Valéria D. Feltrim, Diego Bertolini, and Yandre M. G. Costa. 2022. A Multimodal Approach for Multi-Label Movie Genre Classification. Multimedia Tools Appl. 81, 14 (2022), 19071–19096.

Gonçalo Marques, Marcos Aurélio Domingues, Thibault Langlois, and Fabien Gouyon. 2011. Three Current Issues In Music Autotagging. In Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011, Miami, Florida, USA, 2011. 795–800.

Thomas M. Mitchell. 1997. Machine Learning (1 ed.). McGraw-Hill, Inc., USA.

Elena Montañes, Robin Senge, Jose Barranquero, José Ramón Quevedo, Juan José del Coz, and Eyke Hüllermeier. 2014. Dependent binary relevance models for multi-label classification. Pattern Recognition 47, 3 (2014), 1494–1508.

Rodrigo Mansueli Nunes. 2021. Explorando stacking na classificação automática de textos multirrótulos. Master’s thesis. Universidade Estadual de Maringá. [link].

H. Peng, J. Li, S. Wang, L. Wang, Q. Gong, R. Yang, B. Li, P. Yu, and L. He. 2019. Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification. IEEE Transactions on Knowledge and Data Engineering (2019), 1–1.

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1532–1543.

Giuseppe Portolese, Marcos Aurélio Domingues, and Valéria Delisandra Feltrim. 2019. Exploring Textual Features for Multi-label Classification of Portuguese Film Synopses. In Progress in Artificial Intelligence, 19th EPIA Conference on Artificial Intelligence, EPIA 2019, Vila Real, Portugal, 2019, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 11805). Springer, 669–681.

Giuseppe Portolese and Valéria Feltrin. 2018. On the Use of Synopsis-based Features for Film Genre Classification. In Anais do XV Encontro Nacional de Inteligência Artificial e Computacional (São Paulo). SBC, Porto Alegre, RS, Brasil, 892–902.

David Martin Powers. 2011. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. Journal of Machine Learning Technologies 2 (2011), 37–63.

J. Ross Quinlan. 1986. Induction of Decision Trees. Machine Learning 1, 1 (1986), 81–106.

Raul Rojas. 1996. Neural networks : a systematic introduction. Springer-Verlag, Berlin New York.

Fabrizio Sebastiani. 2002. Machine Learning in Automated Text Categorization. ACM Comput. Surv. 34, 1 (2002), 1–47.

Muhammad Atif Tahir, Josef Kittler, and Ahmed Bouridane. 2016. Multi-label classification using stacked spectral kernel discriminant analysis. Neurocomputing 171 (2016), 127–137.

Pang-Ning Tan, Michael S. Steinbach, and Vipin Kumar. 2005. Introduction to Data Mining. Addison-Wesley.

G. Tsoumakas, I. Katakis, and I. Vlahavas. 2011. Random k-Labelsets for Multilabel Classification. IEEE Transactions on Knowledge and Data Engineering 23, 7 (2011), 1079–1089.

Ran Wang, Robert Ridley, Xiao Su, Weiguang Qu, and Xinyu Dai. 2021. A novel reasoning mechanism for multi-label text classification. Information Processing & Management 58, 2 (2021), 102441.

Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2016. Data Mining, Fourth Edition: Practical Machine Learning Tools and Techniques (4th ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Yuelong Xia, Ke Chen, and Yun Yang. 2021. Multi-label classification with weighted classifier selection and stacked ensemble. Information Sciences 557 (2021), 421–442.
Como Citar

Selecione um Formato
MANSUELI, Rodrigo; DOMINGUES, Marcos Aurélio; FELTRIM, Valéria Delisandra. Improving Multilabel Text Classification with Stacking and Recurrent Neural Networks. In: SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 28. , 2022, Curitiba. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2022 . p. 125-130.