Avaliação Automática de respostas discursivas curtas baseado em três dimensões linguísticas
Resumo
Com o crescimento do uso de ambientes virtuais cresce a necessidade de um sistema avaliador automático para respostas discursivas. Este trabalho propõem um método para avaliação automática de respostas discursivas curtas baseado numa arquitetura de aprendizagem de máquina de 5 etapas. O método preditivo é baseado na coleta de features (140) de similaridade entre textos numa taxonomia de três dimensões linguísticas: léxico, sintático e semântico. Como resultado obtivemos kappa quadrático 0.72 sistema x humano (SxH) contra 0.94 humano x humano (HxH) para a prova de Biologia e uma acurácia 0.76 SxH contra 0.58 HxH para a prova de Geografia.
Palavras-chave:
avaliação automática, respostas discursivas, aprendizagem de máquina, similaridade de textos, dimensões linguísticas
Referências
Alencar, L. F. (2010) “Aelius: uma ferramenta para anotação automática de corpora usando o NLTK”, Anais do IX Encontro de Linguística de Corpus, PUCRS, Porto Alegre, v. 8.
Burrows, S., Gurevych, I. and Stein, B. (2015) “The eras and trends of automatic short answer grading”, International Journal of Artificial Intelligence in Education, v. 25, n. 1, p. 60-117.
Burstein, J. et al. (1998) “Automated scoring using a hybrid feature identification technique”, In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 206–210. Association for Computational Linguistics.
Fernández-Delgado, M. et al. (2014) “Do we need hundreds of classifiers to solve real world classification problems?”. The Journal of Machine Learning Research, v. 15, n. 1, p. 3133-3181.
Fleiss, J. L. and Cohen, J. (1973) “The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability”, Educational and psychological measurement, v. 33, n. 3, p. 613-619.
Galhardi, L. et al. (2018) “Portuguese Automatic Short Answer Grading”. In: Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE). p. 1373.
Gomaa, W. H. and Fahmy, A. A. (2014). “Automatic scoring for answers to arabic test questions”. Computer Speech & Language, 28(4):833–857.
Haley, D. T. et al. (2007) “Seeing the whole picture: evaluating automated assessment systems”. Innovation in Teaching and Learning in Information and Computer Sciences, v. 6, n. 4, p. 203-224.
Landis, J. R. and Koch, G. (1977) “The measurement of observer agreement for categorical data”. biometrics, p. 159-174.
Leacock, C. and Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37(4):389–405.
Learning, V. (2000). “A study of expert scoring and intellimetric scoring accuracy for dimensional scoring of grade 11 student writing responses” (rb-397). Newtown, PA: Vantage Learning.
Mohler, M. and Mihalcea, R. (2009). “Text-to-text semantic similarity for automatic short answer grading”. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 567–575. Association for Computational Linguistics.
Page, E. B. (1966). “The imminence of... grading essays by computer”. The Phi Delta Kappan, v. 47, n. 5, p. 238-243.
Palma, D. and Atkinson, J. (2018) “Coherence-Based Automatic Essay Assessment”. IEEE Intelligent Systems, v. 33, n. 5, p. 26-36.
Pérez, D., Alfonseca, E., Rodr??guez, P., Gliozzo, A., Strapparava, C., and Magnini, B. (2005). “About the effects of combining latent semantic analysis with natural language processing techniques for free-text assessment”. Revista signos, 38(59):325–343.
Pribadi, F. S., Adji, T. B., Permanasari, A. E., Mulwinda, A., and Utomo, A. B. (2017). “Automatic short answer scoring using words overlapping methods”. In AIP Conference Proceedings, volume 1818, page 020042. AIP Publishing.
Rababah, H. e Al-Taani, A. T. (2017) “An automated scoring approach for Arabic short answers essay questions”. In: 8th International Conference on Information Technology (ICIT). IEEE, p. 697-702.
Rodrigues, F. and Araújo, L. (2012) “Automatic Assessment of Short Free Text Answers”. In: CSEDU (2). p. 50-57.
Vajjalla, S. (2018) “Automated assessment of non-native learner essays: Investigating the role of linguistic features”. International Journal of Artificial Intelligence in Education, v. 28, n. 1, p. 79-105.
Zupanc, K. and Bosnic, Z. (2017) “Automated essay evaluation with semantic analysis”. Knowledge-Based Systems, v. 120, p. 118-132.
Burrows, S., Gurevych, I. and Stein, B. (2015) “The eras and trends of automatic short answer grading”, International Journal of Artificial Intelligence in Education, v. 25, n. 1, p. 60-117.
Burstein, J. et al. (1998) “Automated scoring using a hybrid feature identification technique”, In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 206–210. Association for Computational Linguistics.
Fernández-Delgado, M. et al. (2014) “Do we need hundreds of classifiers to solve real world classification problems?”. The Journal of Machine Learning Research, v. 15, n. 1, p. 3133-3181.
Fleiss, J. L. and Cohen, J. (1973) “The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability”, Educational and psychological measurement, v. 33, n. 3, p. 613-619.
Galhardi, L. et al. (2018) “Portuguese Automatic Short Answer Grading”. In: Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE). p. 1373.
Gomaa, W. H. and Fahmy, A. A. (2014). “Automatic scoring for answers to arabic test questions”. Computer Speech & Language, 28(4):833–857.
Haley, D. T. et al. (2007) “Seeing the whole picture: evaluating automated assessment systems”. Innovation in Teaching and Learning in Information and Computer Sciences, v. 6, n. 4, p. 203-224.
Landis, J. R. and Koch, G. (1977) “The measurement of observer agreement for categorical data”. biometrics, p. 159-174.
Leacock, C. and Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37(4):389–405.
Learning, V. (2000). “A study of expert scoring and intellimetric scoring accuracy for dimensional scoring of grade 11 student writing responses” (rb-397). Newtown, PA: Vantage Learning.
Mohler, M. and Mihalcea, R. (2009). “Text-to-text semantic similarity for automatic short answer grading”. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 567–575. Association for Computational Linguistics.
Page, E. B. (1966). “The imminence of... grading essays by computer”. The Phi Delta Kappan, v. 47, n. 5, p. 238-243.
Palma, D. and Atkinson, J. (2018) “Coherence-Based Automatic Essay Assessment”. IEEE Intelligent Systems, v. 33, n. 5, p. 26-36.
Pérez, D., Alfonseca, E., Rodr??guez, P., Gliozzo, A., Strapparava, C., and Magnini, B. (2005). “About the effects of combining latent semantic analysis with natural language processing techniques for free-text assessment”. Revista signos, 38(59):325–343.
Pribadi, F. S., Adji, T. B., Permanasari, A. E., Mulwinda, A., and Utomo, A. B. (2017). “Automatic short answer scoring using words overlapping methods”. In AIP Conference Proceedings, volume 1818, page 020042. AIP Publishing.
Rababah, H. e Al-Taani, A. T. (2017) “An automated scoring approach for Arabic short answers essay questions”. In: 8th International Conference on Information Technology (ICIT). IEEE, p. 697-702.
Rodrigues, F. and Araújo, L. (2012) “Automatic Assessment of Short Free Text Answers”. In: CSEDU (2). p. 50-57.
Vajjalla, S. (2018) “Automated assessment of non-native learner essays: Investigating the role of linguistic features”. International Journal of Artificial Intelligence in Education, v. 28, n. 1, p. 79-105.
Zupanc, K. and Bosnic, Z. (2017) “Automated essay evaluation with semantic analysis”. Knowledge-Based Systems, v. 120, p. 118-132.
Publicado
11/11/2019
Como Citar
SIROTHEAU, Silvério; DOS SANTOS, João Carlos; FAVERO, Eloi; NEGRÃO, Simone.
Avaliação Automática de respostas discursivas curtas baseado em três dimensões linguísticas. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 30. , 2019, Brasília/DF.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 1551-1560.
DOI: https://doi.org/10.5753/cbie.sbie.2019.1551.
