CoNLL Dependency Parser: Extrinsic Evaluation through the Open Information Extraction task
Resumo
Dependency Parsers (DP) are parsers that analyze dependencies between words in a sentence. Currently, dependency parser evaluation is a problem whose solutions are not well defined in the scientific community. Although the DP intrinsic metrics are the foremost choice of evaluation, extrinsic evaluation enables a different evaluation based on a downstream. Different results of DP can vary according to the domain task. Thus, this work applies an Open Information Extraction (OIE) method in Portuguese to provide an extrinsic evaluation of a set of CONLL Dependency Parsers. Our results demonstrate that there is a difference in the evaluation of Dependency Parsers considering a particular task.
Palavras-chave:
dependency parser, extrinsic evaluation, open information extraction
Referências
Banko, M., Cafarella, M. J., Soderland, S., Broadhead, M., and Etzioni, O. Open information extraction from the web. In IJCAI. Vol. 7. pp. 2670–2676, 2007.
Buchholz, S. and Marsi, E. CoNLL-x shared task on multilingual dependency parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X). Association for Computational Linguistics, New York City, pp. 149–164, 2006.
Carroll, J., Briscoe, T., and Sanfilippo, A. Parser evaluation: a survey and a new proposal, 1998.
Duthoo, E. and Mesnard, O. CEA LIST: Processing low-resource languages for CoNLL 2018. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 34–44, 2018.
Fares, M., Oepen, S., Øvrelid, L., Björne, J., and Johansson, R. The 2018 shared task on extrinsic parser evaluation: On the downstream utility of English universal dependency parsers. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 22–33, 2018.
Gamallo, P. and Garcia, M. Linguakit: uma ferramenta multilingue para a análise linguística e a extração de informação. Linguamática 9 (1): 19–28, 2017.
Gamallo, P. and Garcia, M. Task-oriented evaluation of dependency parsing with open information extraction. In Computational Processing of the Portuguese Language. Springer International Publishing, Cham, pp. 77–82, 2018.
Gamallo, P., Garcia, M., and Fernández-Lanza, S. Dependency-based open information extraction. In Proceedings of the Joint Workshop on Unsupervised and Semi-Supervised Learning in NLP. Association for Computational Linguistics, pp. 10–18, 2012.
Glauber, R., Claro, D., and Oliveira, L. Dependency parser on open information extraction for portuguese texts -dptoie and dependentie on iberlef. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019)
co-located with 35th Conference of the Spanish Society for Natural Language Processing (SEPLN 2019). Vol. 2421. CEUR Workshops, pp. 442–448, 2019.
Jones, K. S. Towards better nlp system evaluation. In HUMAN LANGUAGE TECHNOLOGY: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
Kanerva, J., Ginter, F., Miekka, N., Leino, A., and Salakoski, T. Turku neural parser pipeline: An end-to-end system for the CoNLL 2018 shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 133–142, 2018.
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19 (2): 313–330, 1993.
Oepen, S., Øvrelid, L., Björne, J., Johansson, R., Lapponi, E., Ginter, F., and Velldal, E. The 2017 shared task on extrinsic parser evaluation. towards a reusable community infrastructure. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy. pp. 1–16, 2017.
Qi, P., Dozat, T., Zhang, Y., and Manning, C. D. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 160–170, 2018.
Rybak, P. and Wróblewska, A. Semi-supervised neural system for tagging, parsing and lematization. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 45–54, 2018.
Tang, G. Dependency viewer, 2012.
Wanderley, M., Claro, D. B., Souza, M., and de Oliveira, L. Avaliaçâo extrínseca de analisadores de dependência através da extraçâo de informaçâo aberta. In Proceedings of the Symposium in Information and Human Language Technology. Symposium in Information and Human Language Technology, pp. 171–180, 2019.
Zeman, D., Hajič, J., Popel, M., Potthast, M., Straka, M., Ginter, F., Nivre, J., and Petrov, S. Conll 2018 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. pp. 1–21, 2018.
Buchholz, S. and Marsi, E. CoNLL-x shared task on multilingual dependency parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X). Association for Computational Linguistics, New York City, pp. 149–164, 2006.
Carroll, J., Briscoe, T., and Sanfilippo, A. Parser evaluation: a survey and a new proposal, 1998.
Duthoo, E. and Mesnard, O. CEA LIST: Processing low-resource languages for CoNLL 2018. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 34–44, 2018.
Fares, M., Oepen, S., Øvrelid, L., Björne, J., and Johansson, R. The 2018 shared task on extrinsic parser evaluation: On the downstream utility of English universal dependency parsers. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 22–33, 2018.
Gamallo, P. and Garcia, M. Linguakit: uma ferramenta multilingue para a análise linguística e a extração de informação. Linguamática 9 (1): 19–28, 2017.
Gamallo, P. and Garcia, M. Task-oriented evaluation of dependency parsing with open information extraction. In Computational Processing of the Portuguese Language. Springer International Publishing, Cham, pp. 77–82, 2018.
Gamallo, P., Garcia, M., and Fernández-Lanza, S. Dependency-based open information extraction. In Proceedings of the Joint Workshop on Unsupervised and Semi-Supervised Learning in NLP. Association for Computational Linguistics, pp. 10–18, 2012.
Glauber, R., Claro, D., and Oliveira, L. Dependency parser on open information extraction for portuguese texts -dptoie and dependentie on iberlef. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019)
co-located with 35th Conference of the Spanish Society for Natural Language Processing (SEPLN 2019). Vol. 2421. CEUR Workshops, pp. 442–448, 2019.
Jones, K. S. Towards better nlp system evaluation. In HUMAN LANGUAGE TECHNOLOGY: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
Kanerva, J., Ginter, F., Miekka, N., Leino, A., and Salakoski, T. Turku neural parser pipeline: An end-to-end system for the CoNLL 2018 shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 133–142, 2018.
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19 (2): 313–330, 1993.
Oepen, S., Øvrelid, L., Björne, J., Johansson, R., Lapponi, E., Ginter, F., and Velldal, E. The 2017 shared task on extrinsic parser evaluation. towards a reusable community infrastructure. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy. pp. 1–16, 2017.
Qi, P., Dozat, T., Zhang, Y., and Manning, C. D. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 160–170, 2018.
Rybak, P. and Wróblewska, A. Semi-supervised neural system for tagging, parsing and lematization. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, Brussels, Belgium, pp. 45–54, 2018.
Tang, G. Dependency viewer, 2012.
Wanderley, M., Claro, D. B., Souza, M., and de Oliveira, L. Avaliaçâo extrínseca de analisadores de dependência através da extraçâo de informaçâo aberta. In Proceedings of the Symposium in Information and Human Language Technology. Symposium in Information and Human Language Technology, pp. 171–180, 2019.
Zeman, D., Hajič, J., Popel, M., Potthast, M., Straka, M., Ginter, F., Nivre, J., and Petrov, S. Conll 2018 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. pp. 1–21, 2018.
Publicado
20/10/2020
Como Citar
BAIA, Jardel; PRATES, Arley; CLARO, Daniela.
CoNLL Dependency Parser: Extrinsic Evaluation through the Open Information Extraction task. In: SYMPOSIUM ON KNOWLEDGE DISCOVERY, MINING AND LEARNING (KDMILE), 8. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 193-200.
ISSN 2763-8944.
DOI: https://doi.org/10.5753/kdmile.2020.11976.
