Improving Multilabel Text Classification with Stacking and Recurrent Neural Networks

• Rodrigo Mansueli UEM
• Marcos Aurélio Domingues UEM
• Valéria Delisandra Feltrim UEM

Resumo

Multilabel text classification can be defined as a mapping function that categorizes a text in natural language into one or more labels defined by the scope of a problem. In this work we propose an architecture of stacked classifiers for multilabel text classification. The proposed models use an LSTM recurrent neural network in the first stage of the stack and different multilabel classifiers in the second stage. We evaluated our proposal in two datasets well-known in the literature (TMDB and EUR-LEX Subject Matters), and the results showed that the proposed stack consistently outperforms the baselines.

Palavras-chave: Machine Learning, Multilabel Classification, Stacking, Recurrent Neural Network

Referências

Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. 1992. A Training Algorithm for Optimal Margin Classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory (Pittsburgh, Pennsylvania, USA) (COLT ’92). Association for Computing Machinery, New York, NY, USA, 144–152. https://doi.org/10.1145/130385.130401

Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1724–1734. https://doi.org/10.3115/v1/D14-1179

Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks. Machine Learning 20, 3 (1995), 273–297. https://doi.org/10.1007/BF00994018

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423

Katti Faceli, Ana Carolina Lorena, João Gama, and André C. P. L. F de Carvalho. 2019. Inteligência Artificial - Uma Abordagem de Aprendizado de Máquina (3 ed.).Grupo Gen - LTC.

Francisco Herrera, Francisco Charte, Antonio J. Rivera, and María J. del Jesus. 2016. Multilabel Classification. Springer International Publishing.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

Anwesha Law and Ashish Ghosh. 2019. Multi-label classification using a cascade of stacked autoencoder and extreme learning machines. Neurocomputing 358 (2019), 222–234. https://doi.org/10.1016/j.neucom.2019.05.051

Rémi Lebret and Ronan Collobert. 2014. Word Embeddings through Hellinger PCA. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics,Gothenburg, Sweden, 482–490. https://doi.org/10.3115/v1/E14-1051

Eneldo Loza Mencía and Johannes Fürnkranz. 2010. Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain. In Semantic Processing of Legal Texts: Where the Language of Law Meets the Law of Language. Springer Berlin Heidelberg, Berlin, Heidelberg, 192–215. https://doi.org/10.1007/978-3-642-12837-0_11

Eneldo Loza Mencía and Frederik Janssen. 2016. Learning rules for multi-label classification: a stacking and a separate-and-conquer approach. Machine Learning 105, 1 (2016), 77–126. https://doi.org/10.1007/s10994-016-5552-1

Rafael B. Mangolin, Rodolfo M. Pereira, Alceu S. Britto, Carlos N. Silla, Valéria D. Feltrim, Diego Bertolini, and Yandre M. G. Costa. 2022. A Multimodal Approach for Multi-Label Movie Genre Classification. Multimedia Tools Appl. 81, 14 (2022), 19071–19096. https://doi.org/10.1007/s11042-020-10086-2

Gonçalo Marques, Marcos Aurélio Domingues, Thibault Langlois, and Fabien Gouyon. 2011. Three Current Issues In Music Autotagging. In Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011, Miami, Florida, USA, 2011. 795–800. http://ismir2011.ismir.net/papers/OS10-1.pdf

Thomas M. Mitchell. 1997. Machine Learning (1 ed.). McGraw-Hill, Inc., USA.

Elena Montañes, Robin Senge, Jose Barranquero, José Ramón Quevedo, Juan José del Coz, and Eyke Hüllermeier. 2014. Dependent binary relevance models for multi-label classification. Pattern Recognition 47, 3 (2014), 1494–1508. https://doi.org/10.1016/j.patcog.2013.09.029

Rodrigo Mansueli Nunes. 2021. Explorando stacking na classificação automática de textos multirrótulos. Master’s thesis. Universidade Estadual de Maringá. [link].

H. Peng, J. Li, S. Wang, L. Wang, Q. Gong, R. Yang, B. Li, P. Yu, and L. He. 2019. Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification. IEEE Transactions on Knowledge and Data Engineering (2019), 1–1. https://doi.org/10.1109/TKDE.2019.2959991

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1532–1543. https://doi.org/10.3115/v1/D14-1162

Giuseppe Portolese, Marcos Aurélio Domingues, and Valéria Delisandra Feltrim. 2019. Exploring Textual Features for Multi-label Classification of Portuguese Film Synopses. In Progress in Artificial Intelligence, 19th EPIA Conference on Artificial Intelligence, EPIA 2019, Vila Real, Portugal, 2019, Proceedings, Part II (Lecture Notes in Computer Science, Vol. 11805). Springer, 669–681. https://doi.org/10.1007/978-3-030-30244-3_55

Giuseppe Portolese and Valéria Feltrin. 2018. On the Use of Synopsis-based Features for Film Genre Classification. In Anais do XV Encontro Nacional de Inteligência Artificial e Computacional (São Paulo). SBC, Porto Alegre, RS, Brasil, 892–902. https://doi.org/10.5753/eniac.2018.4476

David Martin Powers. 2011. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. Journal of Machine Learning Technologies 2 (2011), 37–63.

J. Ross Quinlan. 1986. Induction of Decision Trees. Machine Learning 1, 1 (1986), 81–106. https://doi.org/10.1023/A:1022643204877

Raul Rojas. 1996. Neural networks : a systematic introduction. Springer-Verlag, Berlin New York.

Fabrizio Sebastiani. 2002. Machine Learning in Automated Text Categorization. ACM Comput. Surv. 34, 1 (2002), 1–47. https://doi.org/10.1145/505282.505283

Muhammad Atif Tahir, Josef Kittler, and Ahmed Bouridane. 2016. Multi-label classification using stacked spectral kernel discriminant analysis. Neurocomputing 171 (2016), 127–137. https://doi.org/10.1016/j.neucom.2015.06.023

Pang-Ning Tan, Michael S. Steinbach, and Vipin Kumar. 2005. Introduction to Data Mining. Addison-Wesley.

G. Tsoumakas, I. Katakis, and I. Vlahavas. 2011. Random k-Labelsets for Multilabel Classification. IEEE Transactions on Knowledge and Data Engineering 23, 7 (2011), 1079–1089. https://doi.org/10.1109/TKDE.2010.164

Ran Wang, Robert Ridley, Xiao Su, Weiguang Qu, and Xinyu Dai. 2021. A novel reasoning mechanism for multi-label text classification. Information Processing & Management 58, 2 (2021), 102441. https://doi.org/10.1016/j.ipm.2020.102441

Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2016. Data Mining, Fourth Edition: Practical Machine Learning Tools and Techniques (4th ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Yuelong Xia, Ke Chen, and Yun Yang. 2021. Multi-label classification with weighted classifier selection and stacked ensemble. Information Sciences 557 (2021), 421–442. https://doi.org/10.1016/j.ins.2020.06.017