Winograd Schemas in Portuguese

• Gabriela de Melo Universidade de São Paulo
• Vinicius Imaizumi Universidade de São Paulo
• Fábio Cozman Universidade de São Paulo

DOI: https://doi.org/10.5753/eniac.2019.9334

Resumo

The Winograd Schema Challenge has become a common benchmark for question answering and natural language processing. The original set of Winograd Schemas was created in English; in order to stimulate the development of Natural Language Processing in Portuguese, we have developed a set of Winograd Schemas in Portuguese. We have also adapted solutions proposed for the English-based version of the challenge so as to have an initial baseline for its Portuguese-based version; to do so, we created a language model for Portuguese based on a set of Wikipedia documents.

Palavras-chave: Applications of Artificial Intelligence, Machine Learning, Natural Language Processing, Artificial Neural Networks, Deep Learning

Referências

Amsili, P. and Seminck, O. (2017). A Google-proof collection of French Winograd schemas. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 24–29, Valencia, Spain. Association for Computational Linguistics.

Bailey, D., Harrison, A., Lierler, Y., Lifschitz, V., and Michael, J. (2015). The winograd schema challenge and reasoning about correlation. In Working Notes of the Symposium on Logical Formalizations of Commonsense Reasoning. AAAI Press.

Bender, D. (2015). Establishing a human baseline for the winograd schema challenge. In Proceedings of the 26th Modern AI and Cognitive Science Conference 2015, Greensboro, NC, USA, April 25-26, 2015., pages 39–45.

Davis, E. (2016). Winograd schemas and machine translation. CoRR, abs/1608.01884.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Emami, A., De La Cruz, N., Trischler, A., Suleman, K., and Cheung, J. C. K. (2018). A knowledge hunting framework for common sense reasoning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1949– 1958, Brussels, Belgium. Association for Computational Linguistics.

Inan, H., Khosravi, K., and Socher, R. (2016). Tying word vectors and word classifiers: A loss framework for language modeling. CoRR, abs/1611.01462.

Kocijan, V., Cretu, A., Camburu, O., Yordanov, Y., and Lukasiewicz, T. (2019). A surprisingly robust trick for winograd schema challenge. CoRR, abs/1905.06290.

Levesque, H. J., Davis, E., and Morgenstern, L. (2012). The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, pages 552–561. AAAI Press.

Liu, Q., Jiang, H., Ling, Z.-H., Zhu, X., Wei, S., and Hu, Y. (2016). Combing context and commonsense knowledge through neural networks for solving winograd schema problems. CoRR, abs/1611.04146.

Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2016). Pointer sentinel mixture models. CoRR, abs/1609.07843.

Opitz, J. and Frank, A. (2018). Addressing the Winograd schema challenge as a sequence ranking task. In Proceedings of the First International Workshop on Language Cognition and Computational Models, pages 41–52, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Peng, H., Khashabi, D., and Roth, D. (2015). Solving hard coreference problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 809–819, Denver, Colorado. Association for Computational Linguistics.

Press, O. and Wolf, L. (2017). Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163, Valencia, Spain. Association for Computational Linguistics.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8).

Rahman, A. and Ng, V. (2012). Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association for Computational Linguistics.

Ruan, Y., Zhu, X., Ling, Z., Shi, Z., Liu, Q., and Wei, S. (2019). Exploring unsupervised pretraining and sentence structure modelling for winograd schema challenge. CoRR, abs/1904.09705.

Schüller, P. (2014). Tackling winograd schemas by formalizing relevance theory in knowedge graphs. In Proceedings of the Fourteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’14, pages 358–367. AAAI Press.

Sharma, A., Vo, N. H., Aditya, S., and Baral, C. (2015). Towards addressing the winograd schema challenge: Building and using a semantic parser and a knowledge hunting module. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI’15, pages 1319–1325. AAAI Press.

Trichelair, P., Emami, A., Cheung, J. C. K., Trischler, A., Suleman, K., and Diaz, F. (2018). On the evaluation of common-sense reasoning in natural language understanding. CoRR, abs/1811.01778.

Trinh, T. H. and Le, Q. V. (2018). A simple method for commonsense reasoning. CoRR, abs/1806.02847.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.