Investigating Methods to Detect Off-Topic Essays - Silva, Joyce M.; Anchiêta, Rafael T.; Sousa, Rogério F. de; Moura, Raimundo S. (Anais da Brazilian Conference on Intelligent Systems (BRACIS), 2024)
Automated Essay Scoring is one of the most important educational applications of natural language processing. It helps teachers with automatic assessments, providing a cheaper, faster, and more deterministic approach than humans when scoring essays. Nevertheless, off-topic essays pose challenges in this area, causing an automated grader to overestimate the score of an essay that does not adhere to a proposed topic. Thus, detecting off-topic essays is important for dealing with unrelated text responses to a given topic. This paper explored approaches based on handcrafted features to feed supervised machine-learning algorithms, tuning a BERT model, and prompt engineering with a large language model. We assessed these strategies in a public corpus of Portuguese essays, achieving the best result using a fine-tuned BERT model with a 75% balanced accuracy. Furthermore, this strategy was able to identify low-quality essays.