Off-Topic Essay Detection: A Systematic Review
Resumo
Essays are widely used for learning assessment in the educational context. Commercial solutions for automated essay scoring have shown promising results, but vulnerability to fraud is still criticized in the scientific community. An off-topic essay detection tool can be used to increase the reliability of automated essay scoring systems and to generate feedback to students. In this context, this paper presents a systematic review of the literature on automatic detection of off-topic essays. We describe the techniques and resources, the corpora and the performance of existing approaches. The results found indicate some gaps and deficiencies in the existing literature, including the need to reduce error rates and to use validation sets based on real examples of off-topic essays.
Palavras-chave:
Essay Detection, Automated Assessment, Off-Topic Essays
Referências
Chen, J. and Zhang, M. (2016). Identifying Useful Features to Detect Off-Topic Essays in Automated Scoring Without Using Topic-Specific Training Essays. Springer Proceedings in Mathematics and Statistics, v. 140, August, p. 315–326.
Higgins, D., Burstein, J. and Attali, Y. (2006). Identifying off-topic student essays without topic-specific training data. Natural Language Engineering, v. 12, n. 2.
Higgins, D. and Heilman, M. (2014). Managing what we can measure: Quantifying the susceptibility of automated scoring systems to gaming behavior. Educational Measurement: Issues and Practice, v. 33, n. 3, p. 36–46.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report.
Li, Y. and Yan, Y. (2012). An effective automated essay scoring system using support vector regression. Proceedings-2012 5th International Conference on Intelligent Computation Technology and Automation, ICICTA 2012, p. 65–68.
Louis, A. and Higgins, D. (2010). Off-topic essay detection using short prompt texts. NAACLHLT2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, June, p. 92–95.
Persing, I. and Ng, V. (2014). Modeling Prompt Adherence in Student Essays. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, June, p. 1534–1543.
Santos, J. J., Paiva, R. and Bittencourt, I. I. (2016). Lexical-Syntactic Evaluation of written activities based on Genetic Algorithm and Natural Language Processing: An experiment on ENEM (in Portuguese). Brazilian Journal of Computers in Education, v. 24, n. 2, p. 92–107.
Silva, A. C., Barbosa, W., Rodrigues, C. L. and Ferreira, D. (2016). Technologies for teaching DHI students using Libras: a systematic review. Brazilian Symposium on Computers in Education, v. 27, n. 1, p. 747–756.
Higgins, D., Burstein, J. and Attali, Y. (2006). Identifying off-topic student essays without topic-specific training data. Natural Language Engineering, v. 12, n. 2.
Higgins, D. and Heilman, M. (2014). Managing what we can measure: Quantifying the susceptibility of automated scoring systems to gaming behavior. Educational Measurement: Issues and Practice, v. 33, n. 3, p. 36–46.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report.
Li, Y. and Yan, Y. (2012). An effective automated essay scoring system using support vector regression. Proceedings-2012 5th International Conference on Intelligent Computation Technology and Automation, ICICTA 2012, p. 65–68.
Louis, A. and Higgins, D. (2010). Off-topic essay detection using short prompt texts. NAACLHLT2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, June, p. 92–95.
Persing, I. and Ng, V. (2014). Modeling Prompt Adherence in Student Essays. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, June, p. 1534–1543.
Santos, J. J., Paiva, R. and Bittencourt, I. I. (2016). Lexical-Syntactic Evaluation of written activities based on Genetic Algorithm and Natural Language Processing: An experiment on ENEM (in Portuguese). Brazilian Journal of Computers in Education, v. 24, n. 2, p. 92–107.
Silva, A. C., Barbosa, W., Rodrigues, C. L. and Ferreira, D. (2016). Technologies for teaching DHI students using Libras: a systematic review. Brazilian Symposium on Computers in Education, v. 27, n. 1, p. 747–756.
Publicado
30/10/2017
Como Citar
PASSERO, Guilherme; FERREIRA, Rafael; HAENDCHEN FILHO, Aluizio; SCARANTO DAZZI, Rudimar Luís.
Off-Topic Essay Detection: A Systematic Review. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 28. , 2017, Recife/PE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2017
.
p. 51-60.
DOI: https://doi.org/10.5753/cbie.sbie.2017.51.
