Using Linguistic Resources for Automatic Classification of Feedback Messages

• Anderson Pinheiro Cavalcanti Federal University of Pernambuco / SiDi http://orcid.org/0000-0002-5228-1583
• Rafael Ferreira Mello Federal Rural University of Pernambuco / Cesar School http://orcid.org/0000-0003-3548-9670
• Péricles Miranda Federal Rural University of Pernambuco
• André Nascimento Federal Rural University of Pernambuco
• Fred Freitas Federal University of Pernambuco

DOI: https://doi.org/10.5753/sbie.2021.218481

Abstract

Feedback plays a significant role in a student's learning process. It allows students to identify weaknesses and improve self-regulation. However, studies show that this is an area of great dissatisfaction in higher education. With the increasing number of course participants, providing effective feedback is becoming an increasingly challenging task. Therefore, this article explores the use of automated content analysis to examine instructor-provided feedback based on well-regarded models from the literature that provide good practices and rank feedback at different levels. For this, binary classifiers were trained using the XGBoost algorithm together with linguistic resources that extract characteristics from the feedback messages. The results indicate effective classification performance for both good practices and feedback levels, reaching accuracy values of up to 0.89 in the test set.

Keywords: Feedback, Language Resources, Text Classification

References

Barbosa, A., Ferreira, M., Ferreira Mello, R., Dueire Lins, R., and Gasevic, D.(2021). The impact of automatic text translation on classification of online discussions for social and cognitive presences. In LAK21: 11th International Learning Analytics and Knowledge Conference, pages 77–87.

Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovic, V., and Gasevic, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 605–614.

Batista, R. L., Salgado, N., and Barreto, R. (2018). Avaliação e feedback automático em educação apoiada por tecnologia: Um mapeamento sistemático da literatura. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 29, page 178.

Butler, D. L. and Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational Research, 65(3):245–281.

Camelo, R., Justino, S., and de Mello, R. F. L. (2020). Coh-metrix pt-br: uma api web de análise textual para a educação. In Anais dos Workshops do IX Congresso Brasileiro de Informática na Educação, pages 179–186. SBC

Cavalcanti, A., Rolim, V., Mello, R., and Freitas, F. (2020). Uma análise entre boas práticas de feedback em ambientes virtuais de aprendizagem. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 962–971, Porto Alegre, RS, Brasil. SBC.

Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gasevic, D., and Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2:100027.

Cavalcanti, A. P., de Mello, R. F. L., de Miranda, P. B. C., and de Freitas, F. L. G. (2020). Análise automática de feedback em ambientes de aprendizagem online. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 892–901. SBC.

Cavalcanti, A. P., de Mello, R. F. L., Rolim, V., André, M., Freitas, F., and Gasevic, D. (2019). An analysis of the use of good feedback practices in online learning courses. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), volume 2161, pages 153–157. IEEE.

Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento,A., Freitas, F., and Gasevic, D. (2020). How good is my feedback? a content analysis of written feedback. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge - LAK. ACM.

Chen, T. and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, San Francisco, California, USA. ACM.

Clark, I. (2012). Formative Assessment: Assessment Is for Self-regulated Learning. Educ. Psychol. Rev., 24(2):205–249.

Dweck, C. S. (1999). Self-theories: Their role in motivation, personality, and development. Self-theories: Their role in motivation, personality, and development. Psychology Press, New York, NY, US. Pages: xiii, 195.

Ferguson, P. (2011). Student perceptions of quality feedback in teacher education. Assess Eval High Educ, 36(1): 51–62.

Ferreira, M., Rolim, V., Mello, R. F., Lins, R. D., Chen, G., and Gasevic, D. (2020). Towards automatic content analysis of social presence in transcripts of online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 141–150.

Ferreira-Mello, R., André, M., Pinheiro, A., Costa, E., and Romero, C. (2019). Text mining in education.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(6): e1332.

Friedman, J., Hastie, T., and Tibshirani, R. (2001).The elements of statistical learning, volume 1. Springer series in statistics New York, NY, USA

Hattie, J. and Gan, M. (2011). Instruction based on feedback. In Handbook of research on learning and instruction pages 263–285. Routledge.

Hattie, J. and Timperley, H. (2007). The power of feedback.Review of educational research, 77(1): 81–112.

Laurillard, D. (1993). Rethinking University Teaching: Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. Routledge.

Liu, M., Li, Y., Xu, W., and Liu, L. (2017). Automated Essay Feedback Generation and Its Impact on Revision. IEEE Trans. Learn. Technol., 10(4): 502–513.

Ma, X., Wijewickrema, S., Zhou, S., Zhou, Y., Mhammedi, Z., O’Leary, S., and Bailey, J. (2017). Adversarial Generation of Real-time Feedback with Neural Networks for Simulation-based Training. InProceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 3763–3769, Melbourne, Australia. International Joint Conferences on Artificial Intelligence Organization.

Mani, I. and Zhang, I. (2003). knn approach to unbalanced data distributions: a case study involving information extraction. In Proceedings of Workshop on Learning from Imbalanced Datasets.

McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.

Neuendorf, K. A. (2016).The content analysis guidebook. Sage.

Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2):199–218.

Osakwe, I., Chen, G., Whitelock-Wainwright, A., Gasevic, D., Cavalcanti, A. P., and Mello, R. F. (2021). Towards automated content analysis of feedback: A multi-language study. In Proceedings of the 14th International Conference on Educational Data Mining.

Pan, B. (2018). Application of XGBoost algorithm in hourly PM2.5 concentration pre-diction. IOP Conference Series: Earth and Environmental Science, 113:012127. Publisher: IOP Publishing.

Parikh, A., McReelis, K., and Hodges, B. (2001). Student feedback in problem based learning: a survey of 103 final year students across five ontario medical schools. Medical education, 35(7):632–636.

Price, M., Handley, K., Millar, J., and O’Donovan, B. (2010). Feedback: all that effort, but what is the effect? Assess Eval High Educ, 35(3):277–289

Tausczik, Y. R. and Pennebaker, J. W. (2010). The psychological meaning of words: Liwc and computerized text analysis methods. Journal of language and social psychology, 29(1):24–54.

Villalón, J., Kearney, P., Calvo, R. A., and Reimann, P. (2008). Glosser: Enhanced Feedback for Student Writing Tasks. In 2008 Eighth IEEE International Conference on Advanced Learning Technologies, pages 454–458, Santander, Cantabria, Spain. IEEE

Weaver, M. R. (2006). Do students value feedback? Student perceptions of tutors’ written responses. Assess Eval High Educ, 31(3):379–394.

Xiao, Z., Wang, Y., Fu, K., and Wu, F. (2017). Identifying Different Transportation Modes from Trajectory Data Using Tree-Based Ensemble Classifiers. ISPRS International Journal of Geo-Information, 6(2):57. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.