Essays’ Coherence Analysis Via Entity Grid Approach

• Jessica Oliveira Brito UFES
• Elias de Oliveira UFES

DOI: https://doi.org/10.5753/sbie.2023.235218

Resumo

Coherence analysis is a challenging task, especially when applied to many domains. This paper proposes a strategy that combines Machine Learning and Linguistics to analyze text coherence by understanding entity behavior. It introduces an algorithm that automatically annotates documents based on the Entity Grid Discourse Representation. We defined two datasets, one of academic papers’ abstracts and the other of students’ essays. The annotation technique identifies the influence of grammatical structures on coherence levels and offers a cost-effective solution for coherence analysis. To assess coherence levels Machine Learning methods were used, and the experimental results demonstrate an accuracy of 88% when assessing coherence in abstracts and 74% in essays.

Referências

Althaus, E., Karamanis, N., and Koller, A. (2004). Computing locally coherent discourses. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 399–406.

Alves, J. and Oliveira, E. (2019). Avaliação de juízes: Um modelo estatístico para perfilação de avaliadores. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 199.

Barzilay, R. and Lapata, M. (2008). Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1–34.

Barzilay, R. and Lee, L. (2004). Catching the drift: Probabilistic content models, with applications to generation and summarization. arXiv preprint cs/0405039.

Brito, J., Alves, J., Badue, C., and Oliveira, E. (2021). An architecture for massive essays evaluations. In 2021 16th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–6. IEEE.

da Silva Junior, J. A. (2021). Um avaliador automático de redaçoes. Master’s thesis, Universidade Federal do Espírito Santo.

Elsner, M., Austerweil, J., and Charniak, E. (2007). A unified local and global model for discourse coherence. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 436–443.

Grosz, B. J., Joshi, A. K., and Weinstein, S. (1995). Centering: A framework for modeling the local coherence of discourse.

Lapata, M., Barzilay, R., et al. (2005). Automatic evaluation of text coherence: Models and representations. In Ijcai, volume 5, pages 1085–1090.

Li, J. and Hovy, E. (2014). A model of coherence based on distributed sentence representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2039–2048.

Logeswaran, L., Lee, H., and Radev, D. (2018). Sentence ordering and coherence modeling using recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.

Marinho, J. C., Cordeiro, F., Anchiêta, R. T., and Moura, R. S. (2022). Automated essay scoring: An approach based on enem competencies. In Anais do XIX Encontro Nacional de Inteligência Artificial e Computacional, pages 49–60. SBC.

Oliveira, E., Alves, J., Brito, J., and Pirovani, J. (2021). The influence of ner on the essay grading. In Intelligent Systems Design and Applications: 19th International Conference on Intelligent Systems Design and Applications (ISDA 2019) held December 3-5, 2019 19, pages 162–171. Springer.

Shin, J. and Gierl, M. J. (2022). Evaluating coherence in writing: Comparing the capacity of automated essay scoring technologies. Journal of Applied Testing Technology, 23:04–20.