Interpretability of Attention Mechanisms in a Portuguese-Based Question Answering System about the Blue Amazon

• Stefano Spindola USP http://orcid.org/0000-0002-3604-9346
• Marcos Menon José USP https://orcid.org/0000-0003-4663-4386
• André Seidel Oliveira USP https://orcid.org/0000-0001-6551-6911
• Flávio Nakasato Cação USP https://orcid.org/0000-0003-4771-6009
• Fábio Gagliardi Cozman USP https://orcid.org/0000-0003-4077-4935

DOI: https://doi.org/10.5753/eniac.2021.18302

Resumo

The Brazilian Exclusive Economic Zone, or the "Blue Amazon", with its extensive maritime area, is the primary means of transport for the country's foreign trade and is important due to its oil reserves, gas and other mineral resources, in addition to the significant influence on the Brazilian climate. We have manually built a question answering (QA) dataset based on crawled articles and have applied an off-the-shelf QA system based on a fine-tuned BERTimbau Model, achieving an F1-score of 47.0. More importantly, we explored how the proper visualization of attention weights can support helpful interpretations of the system's answers, which is critical in real environments.

Referências

Bender, E. M. and Friedman, B. (2018). Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604.

Castro, B. M., Brandini, F. P., Dottori, M., and Fortes, J. F. (2017). A Amazônia Azul: recursos e preservação. Revista USP, (113):7.

Devlin, J., Chang, M. W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL HLT 2019 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings of the Conference, 1(Mlm):4171– 4186.

Ghaeini, R., Fern, X. Z., and Tadepalli, P. (2020). Interpreting recurrent and attentionbased neural models: A case study on natural language inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, pages 4952–4957.

Guillou, P. (2021). Portuguese bert base cased qa (question answering), finetuned on squad v1.1.

Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). REALM: retrievalaugmented language model pre-training. CoRR, abs/2002.08909.

Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics.

Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260–270, San Diego, California. Association for Computational Linguistics.

Lee, J., Shin, J. H., and Kim, J. S. (2017). Interactive visualization and manipulation of attention-based neural machine translation. EMNLP 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Proceedings, pages 121–126.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.

Lewis, P., Wu, Y., Liu, L., Minervini, P., Küttler, H., Piktus, A., Stenetorp, P., and Riedel, S. (2021). Paq: 65 million probably-asked questions and what you can do with them.

Li, X., Bing, L., Zhang, W., and Lam, W. (2019). Exploiting BERT for end-to-end aspectbased sentiment analysis. CoRR, abs/1910.00883.

Liu, Y. and Lapata, M. (2019). Text summarization with pretrained encoders. CoRR, abs/1908.08345.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Peter, W. L., and Liu, J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv, 21:1–67.

Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250.

Robertson, S. and Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond, volume 3.

Salas, S., Chuenpagdee, R., Charles, A. T., Seijo, J. C., et al. (2011). Coastal fisheries of Latin America and the Caribbean, volume 544. Food and Agriculture Organization of the United Nations ˆ eRome Rome.

Serrano, S. and Smith, N. A. (2020). Is attention interpretable? ACL 2019 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 2931–2951.

Souza, F., Nogueira, R., and Lotufo, R. (2020a). BERTimbau: Pretrained BERT Models for Brazilian Portuguese. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 12319 LNAI, pages 403–417.

Souza, F., Nogueira, R., and Lotufo, R. (2020b). BERTimbau: Pretrained BERT Models In Cerri, R. and Prati, R. C., editors, Intelligent Systems, for Brazilian Portuguese. pages 403–417, Cham. Springer International Publishing.

Thompson, B. N. and Muggah, R. (2015). The Blue Amazon: Brazil Asserts Its Inuence Across the Atlantic.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 2017-Decem(Nips):5999–6009.

Vig, J. (2019a). A multiscale visualization of attention in the transformer model. ACL 2019 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of System Demonstrations, pages 37–42.

Vig, J. (2019b). Bertviz: A tool for visualizing multi-head self-attention in the bert model.

Wiegreffe, S. and Pinter, Y. (2020). Attention is not not explanation. EMNLP-IJCNLP 2019 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 11–20.

Wiesebron, M. (2013). Amazônia Azul: Pensando a Defesa Do Território Marítimo Brasileiro. AUSTRAL: Brazilian Journal of Strategy & International Relations, 2(3):107–132.