Towards analysis on textual inference at ASSIN-2 dataset

• Felipe O. da Silva USP http://orcid.org/0009-0006-6484-5661
• Giovana Meloni Craveiro USP
• Vinícius F. da Silva USP
• Vinícius João de Barros Vanzin USP

DOI: https://doi.org/10.5753/stil.2023.234210

Resumo

In this article, we conduct a preliminary analysis of different methods to address the Textual Entailment Recognition (RTE) task in Portuguese. We use the ASSIN-2 dataset as a benchmark to evaluate our models. Our work combines various textual representation approaches, including bag of words and word embeddings, with machine learning models. Additionally, we present a rule-based approach. Our highest performance was achieved by the BERTimbau-large model fine-tuned on ASSIN-2, which attained an F1 score of 0.89%, positioning it just 1% below the current state-of-the-art. Our ongoing experiment aims to combine our different approaches to leverage their full potential.

Palavras-chave: textual inference, NLP, natural language inference, recognize textual inference, neural-symbolic

Referências

Bowman, S. and Zhu, X. (2019). Deep learning for natural language inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 6–8, Minneapolis, Minnesota. Association for Computational Linguistics https://aclanthology.org/N19-5002/

Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 https://aclanthology.org/D15-1075/

Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., and Wei, S. (2018). Neural natural language inference models enhanced with external knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2406–2417, Melbourne, Australia. Association for Computational Linguistics. https://aclanthology.org/P18-1224/

Das, B. and Chakraborty, S. (2018). An improved text sentiment classification model using tf-idf and next word negation. arXiv preprint arXiv:1806.06407. https://arxiv.org/abs/1806.06407

Hartmann, N. S., Fonseca, E., Shulby, C. D., Treviso, M. V., Rodrigues, J. S., and Aluísio, S. M. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. arXiv preprint arXiv:1708.06025 https://aclanthology.org/W17-6615/

Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 https://aclanthology.org/P18-1031/

Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. https://arxiv.org/abs/1301.3781

Oliveira da Silva, F., Craveiro, G. M., Siqueira Souza, J. M., Silva, V. F. d., and Vanzin, V. J. d. B. (2023). Natural language inference bow, word embeddings and symbolic experiments at assin-2 dataset. GitHub repository https://github.com/jmssouza/nlp_entailment/

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830 https://arxiv.org/abs/1201.0490

Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. (2018). Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31 [link].

Real, L., Fonseca, E., and Gonçalo Oliveira, H. (2020). The ASSIN 2 Shared Task: A Quick Overview, pages 406–412. [link].

Schuster, M. and Paliwal, K. (1997). Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45:2673 – 2681 [link].

Shlens, J. (2014). A tutorial on principal component analysis. arXiv preprint arXiv:1404.1100. https://arxiv.org/abs/1404.1100

Souza, F., Nogueira, R., and Lotufo, R. (2020). Bertimbau: Pretrained bert models for brazilian portuguese. In Cerri, R. and Prati, R. C., editors, Intelligent Systems, pages 403–417, Cham. Springer International Publishing. [link].

Wagner Filho, J. A., Wilkens, R., Idiart, M., and Villavicencio, A. (2018). The brwac corpus: A new open resource for brazilian portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) https://aclanthology.org/L18-1686/

Williams, A., Nangia, N., and Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112– 1122. Association for Computational Linguistics. https://aclanthology.org/N18-1101/

Zhang, Y., Jin, R., and Zhou, Z.-H. (2010). Understanding bag-of-words model: a statistical framework. International Journal of Machine Learning and Cybernetics, 1(1):43–52 [link].