Automatic Assessment of Essays, in Portuguese, focusing on linguistic attributes of surface and content
Abstract
There is growing need for intelligent environments for distance learning. One of its elements is a system of automatic evaluation of conceptual discursive issues. In this work, we propose a method of automatic evaluation of a test in the Portuguese language based on the refinement of content, coherence and surface statistics features to predict the score of an essay. The accuracy of the system was contrasted with the accuracy measured between two human evaluators (HxH), which resulted in an average error value of 0.91 SxH versus 0.89 HxH and a quadratic kappa accuracy of 0.62 SxH versus 0.52 HxH.This study shows that this technology is reaching maturity for use in environments.
References
Attali, Y. and Burstein, J. (2006). Automated essay scoring with e-rater R v. 2. The Journal of Technology, Learning and Assessment, 4(3).
Burstein, J., Chodorow, M., and Leacock, C. (2004). Automated essay evaluation: The criterion online writing service. Ai Magazine, 25(3):27–27.
Burstein, J., Kukich, K., Wolff, S., Lu, C., and Chodorow, M. (1998). Computer analysis of essays. In NCME Symposium on Automated Scoring.
Carel, M. and Ducrot, O. (2001). O problema do paradoxo em uma semântica argumen- tativa. Lı́nguas e instrumentos lingüı́sticos, 8:33–50.
Cheniti-Belcadhi, L., Braham, R., Henze, N., and Nejdl, W. (2004). A generic framework for assessment in adaptive educational hypermedia. In ICWI, pages 397–404.
Fleiss, J. L. and Cohen, J. (1973). The equivalence of weighted kappa and the intra- class correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613–619.
Foltz, P. W., Streeter, L. A., Lochbaum, K. E., and Landauer, T. K. (2013). Implementa- tion and applications of the intelligent essay assessor. Handbook of automated essay evaluation, pages 68–88.
Grosz, B. J., Weinstein, S., and Joshi, A. K. (1995). Centering: A framework for modeling the local coherence of discourse. Computational linguistics, 21(2):203–225.
Haley, D. T., Thomas, P., De Roeck, A., and Petre, M. (2007). Seeing the whole picture: evaluating automated assessment systems. Innovation in Teaching and Learning in Information and Computer Sciences, 6(4):203–224.
Laham, D., Foltz, P., and Landauer, T. (2000). The intelligent essay assessor. IEEE Intelligent Systems.
Mann, W. C. and Thompson, S. A. (1988). Rhetorical structure theory: Toward a func- tional theory of text organization. Text-Interdisciplinary Journal for the Study of Dis- course, 8(3):243–281.
Matthiessen, C. and Thompson, S. A. (1988). The structure of discourse and ‘subordina- tion’. Clause combining in grammar and discourse, 18:275–329.
Miller, D. I., Talbot, V., Gagnon, M., and Messier, C. (2013). Administration of neuropsy- chological tests using interactive voice response technology in the elderly: validation and limitations. Frontiers in Neurology, 4:107.
Page, E. B. (1966). The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243.
Palma, D. and Atkinson, J. (2018). Coherence-based automatic essay assessment. IEEE Intelligent Systems, 33(5):26–36.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
Rich, C. S., Schneider, M. C., and D’BROT, J. M. (2013). Applications of automated essay evaluation in west virginia. In Handbook of Automated Essay Evaluation, pages 121–145. Routledge.
Schultz, M. T. (2013). The intellimetric automated essay scoring engine-a review and an application to chinese essay scoring. Handbook of automated essay scoring: Current applications and future directions, pages 89–98.
Shermis, M. D. and Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge.
Vajjala, S. (2018). Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education, 28(1):79–105.
Valenti, S., Neri, F., and Cucchiarelli, A. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education: Research, 2:319–330.
Zupanc, K. and Bosnic, Z. (2016). Advances in the field of automated essay evaluation. Informatica, 39(4).
Zupanc, K. and Bosnic, Z. (2017). Automated essay evaluation with semantic analysis. Knowledge-Based Systems, 120:118–132.
Burstein, J., Chodorow, M., and Leacock, C. (2004). Automated essay evaluation: The criterion online writing service. Ai Magazine, 25(3):27–27.
Burstein, J., Kukich, K., Wolff, S., Lu, C., and Chodorow, M. (1998). Computer analysis of essays. In NCME Symposium on Automated Scoring.
Carel, M. and Ducrot, O. (2001). O problema do paradoxo em uma semântica argumen- tativa. Lı́nguas e instrumentos lingüı́sticos, 8:33–50.
Cheniti-Belcadhi, L., Braham, R., Henze, N., and Nejdl, W. (2004). A generic framework for assessment in adaptive educational hypermedia. In ICWI, pages 397–404.
Fleiss, J. L. and Cohen, J. (1973). The equivalence of weighted kappa and the intra- class correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613–619.
Foltz, P. W., Streeter, L. A., Lochbaum, K. E., and Landauer, T. K. (2013). Implementa- tion and applications of the intelligent essay assessor. Handbook of automated essay evaluation, pages 68–88.
Grosz, B. J., Weinstein, S., and Joshi, A. K. (1995). Centering: A framework for modeling the local coherence of discourse. Computational linguistics, 21(2):203–225.
Haley, D. T., Thomas, P., De Roeck, A., and Petre, M. (2007). Seeing the whole picture: evaluating automated assessment systems. Innovation in Teaching and Learning in Information and Computer Sciences, 6(4):203–224.
Laham, D., Foltz, P., and Landauer, T. (2000). The intelligent essay assessor. IEEE Intelligent Systems.
Mann, W. C. and Thompson, S. A. (1988). Rhetorical structure theory: Toward a func- tional theory of text organization. Text-Interdisciplinary Journal for the Study of Dis- course, 8(3):243–281.
Matthiessen, C. and Thompson, S. A. (1988). The structure of discourse and ‘subordina- tion’. Clause combining in grammar and discourse, 18:275–329.
Miller, D. I., Talbot, V., Gagnon, M., and Messier, C. (2013). Administration of neuropsy- chological tests using interactive voice response technology in the elderly: validation and limitations. Frontiers in Neurology, 4:107.
Page, E. B. (1966). The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243.
Palma, D. and Atkinson, J. (2018). Coherence-based automatic essay assessment. IEEE Intelligent Systems, 33(5):26–36.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
Rich, C. S., Schneider, M. C., and D’BROT, J. M. (2013). Applications of automated essay evaluation in west virginia. In Handbook of Automated Essay Evaluation, pages 121–145. Routledge.
Schultz, M. T. (2013). The intellimetric automated essay scoring engine-a review and an application to chinese essay scoring. Handbook of automated essay scoring: Current applications and future directions, pages 89–98.
Shermis, M. D. and Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge.
Vajjala, S. (2018). Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education, 28(1):79–105.
Valenti, S., Neri, F., and Cucchiarelli, A. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education: Research, 2:319–330.
Zupanc, K. and Bosnic, Z. (2016). Advances in the field of automated essay evaluation. Informatica, 39(4).
Zupanc, K. and Bosnic, Z. (2017). Automated essay evaluation with semantic analysis. Knowledge-Based Systems, 120:118–132.
Published
2019-07-12
How to Cite
SIROTHEAU, Silvéiro; SANTOS, João; FAVERO, Eloi.
Automatic Assessment of Essays, in Portuguese, focusing on linguistic attributes of surface and content. In: WORKSHOP ON COMPUTING EDUCATION (WEI), 27. , 2019, Belém.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 255-265.
ISSN 2595-6175.
DOI: https://doi.org/10.5753/wei.2019.6634.
