Avaliação Automática de Ensaios, em português, centrada em atributos linguı́sticos de superfı́cie e de conteúdo
Resumo
Cresce a necessidade de ambientes inteligentes para o ensino a distância. Um dos seus elementos é um sistema de avaliação automática de questões conceituais discursivas. Neste trabalho, propõe-se um método de avaliação automática de ensaio na lı́ngua portuguesa baseado no refinamento de atributos de conteúdo (semânticos), de coerência e estatı́sticos de superfı́cie para predizer a pontuação de um ensaio. A acurácia do sistema (SxH) foi contrastada com a acurácia medida entre dois avaliadores humanos (HxH), o que resultou num valor de erro médio de 0.91 SxH contra 0.89 HxH e numa acurácia kappa quadrático de 0.62 SxH contra 0.52 HxH. Este estudo mostra que esta tecnologia está alcançando maturidade para o uso em ambientes.
Referências
Attali, Y. and Burstein, J. (2006). Automated essay scoring with e-rater R v. 2. The Journal of Technology, Learning and Assessment, 4(3).
Burstein, J., Chodorow, M., and Leacock, C. (2004). Automated essay evaluation: The criterion online writing service. Ai Magazine, 25(3):27–27.
Burstein, J., Kukich, K., Wolff, S., Lu, C., and Chodorow, M. (1998). Computer analysis of essays. In NCME Symposium on Automated Scoring.
Carel, M. and Ducrot, O. (2001). O problema do paradoxo em uma semântica argumen- tativa. Lı́nguas e instrumentos lingüı́sticos, 8:33–50.
Cheniti-Belcadhi, L., Braham, R., Henze, N., and Nejdl, W. (2004). A generic framework for assessment in adaptive educational hypermedia. In ICWI, pages 397–404.
Fleiss, J. L. and Cohen, J. (1973). The equivalence of weighted kappa and the intra- class correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613–619.
Foltz, P. W., Streeter, L. A., Lochbaum, K. E., and Landauer, T. K. (2013). Implementa- tion and applications of the intelligent essay assessor. Handbook of automated essay evaluation, pages 68–88.
Grosz, B. J., Weinstein, S., and Joshi, A. K. (1995). Centering: A framework for modeling the local coherence of discourse. Computational linguistics, 21(2):203–225.
Haley, D. T., Thomas, P., De Roeck, A., and Petre, M. (2007). Seeing the whole picture: evaluating automated assessment systems. Innovation in Teaching and Learning in Information and Computer Sciences, 6(4):203–224.
Laham, D., Foltz, P., and Landauer, T. (2000). The intelligent essay assessor. IEEE Intelligent Systems.
Mann, W. C. and Thompson, S. A. (1988). Rhetorical structure theory: Toward a func- tional theory of text organization. Text-Interdisciplinary Journal for the Study of Dis- course, 8(3):243–281.
Matthiessen, C. and Thompson, S. A. (1988). The structure of discourse and ‘subordina- tion’. Clause combining in grammar and discourse, 18:275–329.
Miller, D. I., Talbot, V., Gagnon, M., and Messier, C. (2013). Administration of neuropsy- chological tests using interactive voice response technology in the elderly: validation and limitations. Frontiers in Neurology, 4:107.
Page, E. B. (1966). The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243.
Palma, D. and Atkinson, J. (2018). Coherence-based automatic essay assessment. IEEE Intelligent Systems, 33(5):26–36.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
Rich, C. S., Schneider, M. C., and D’BROT, J. M. (2013). Applications of automated essay evaluation in west virginia. In Handbook of Automated Essay Evaluation, pages 121–145. Routledge.
Schultz, M. T. (2013). The intellimetric automated essay scoring engine-a review and an application to chinese essay scoring. Handbook of automated essay scoring: Current applications and future directions, pages 89–98.
Shermis, M. D. and Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge.
Vajjala, S. (2018). Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education, 28(1):79–105.
Valenti, S., Neri, F., and Cucchiarelli, A. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education: Research, 2:319–330.
Zupanc, K. and Bosnic, Z. (2016). Advances in the field of automated essay evaluation. Informatica, 39(4).
Zupanc, K. and Bosnic, Z. (2017). Automated essay evaluation with semantic analysis. Knowledge-Based Systems, 120:118–132.
Burstein, J., Chodorow, M., and Leacock, C. (2004). Automated essay evaluation: The criterion online writing service. Ai Magazine, 25(3):27–27.
Burstein, J., Kukich, K., Wolff, S., Lu, C., and Chodorow, M. (1998). Computer analysis of essays. In NCME Symposium on Automated Scoring.
Carel, M. and Ducrot, O. (2001). O problema do paradoxo em uma semântica argumen- tativa. Lı́nguas e instrumentos lingüı́sticos, 8:33–50.
Cheniti-Belcadhi, L., Braham, R., Henze, N., and Nejdl, W. (2004). A generic framework for assessment in adaptive educational hypermedia. In ICWI, pages 397–404.
Fleiss, J. L. and Cohen, J. (1973). The equivalence of weighted kappa and the intra- class correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613–619.
Foltz, P. W., Streeter, L. A., Lochbaum, K. E., and Landauer, T. K. (2013). Implementa- tion and applications of the intelligent essay assessor. Handbook of automated essay evaluation, pages 68–88.
Grosz, B. J., Weinstein, S., and Joshi, A. K. (1995). Centering: A framework for modeling the local coherence of discourse. Computational linguistics, 21(2):203–225.
Haley, D. T., Thomas, P., De Roeck, A., and Petre, M. (2007). Seeing the whole picture: evaluating automated assessment systems. Innovation in Teaching and Learning in Information and Computer Sciences, 6(4):203–224.
Laham, D., Foltz, P., and Landauer, T. (2000). The intelligent essay assessor. IEEE Intelligent Systems.
Mann, W. C. and Thompson, S. A. (1988). Rhetorical structure theory: Toward a func- tional theory of text organization. Text-Interdisciplinary Journal for the Study of Dis- course, 8(3):243–281.
Matthiessen, C. and Thompson, S. A. (1988). The structure of discourse and ‘subordina- tion’. Clause combining in grammar and discourse, 18:275–329.
Miller, D. I., Talbot, V., Gagnon, M., and Messier, C. (2013). Administration of neuropsy- chological tests using interactive voice response technology in the elderly: validation and limitations. Frontiers in Neurology, 4:107.
Page, E. B. (1966). The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243.
Palma, D. and Atkinson, J. (2018). Coherence-based automatic essay assessment. IEEE Intelligent Systems, 33(5):26–36.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
Rich, C. S., Schneider, M. C., and D’BROT, J. M. (2013). Applications of automated essay evaluation in west virginia. In Handbook of Automated Essay Evaluation, pages 121–145. Routledge.
Schultz, M. T. (2013). The intellimetric automated essay scoring engine-a review and an application to chinese essay scoring. Handbook of automated essay scoring: Current applications and future directions, pages 89–98.
Shermis, M. D. and Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge.
Vajjala, S. (2018). Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education, 28(1):79–105.
Valenti, S., Neri, F., and Cucchiarelli, A. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education: Research, 2:319–330.
Zupanc, K. and Bosnic, Z. (2016). Advances in the field of automated essay evaluation. Informatica, 39(4).
Zupanc, K. and Bosnic, Z. (2017). Automated essay evaluation with semantic analysis. Knowledge-Based Systems, 120:118–132.
Publicado
12/07/2019
Como Citar
SIROTHEAU, Silvéiro; SANTOS, João; FAVERO, Eloi.
Avaliação Automática de Ensaios, em português, centrada em atributos linguı́sticos de superfı́cie e de conteúdo. In: WORKSHOP SOBRE EDUCAÇÃO EM COMPUTAÇÃO (WEI), 27. , 2019, Belém.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 255-265.
ISSN 2595-6175.
DOI: https://doi.org/10.5753/wei.2019.6634.
