Multiclass classification for feedback quality analysis

• Hyan H. N. Batista Federal Rural University of Pernambuco
• Anderson Pinheiro Cavalcanti Federal University of Pernambuco / SiDi http://orcid.org/0000-0002-5228-1583
• Péricles Miranda Federal Rural University of Pernambuco http://orcid.org/0000-0002-5767-7544
• André Nascimento Federal Rural University of Pernambuco http://orcid.org/0000-0002-9333-3212
• Rafael Ferreira Mello Federal Rural University of Pernambuco / Center for Advanced Studies and Systems of Recife http://orcid.org/0000-0003-3548-9670

DOI: https://doi.org/10.5753/sbie.2022.225396

Abstract

Feedback is a very important factor in the teaching-learning process and crucial in Distance Education, because, as teachers and students are separated in space and/or time, it is through feedback that the student will understand how their performance is in the classroom. discipline and what are the next steps of learning. There are feedback models in the literature that help the teacher to structure and provide quality feedback to the student. In this work, we use Hattie and Timperley's highly regarded feedback model, which divides feedback into categories (task, task processing, regulation, and personal). It is possible to find in the literature works that analyze feedback automatically based on this model. However, these works use traditional machine learning algorithms and train binary classifiers for each level of feedback. Thus, this work aims to use deep learning algorithms for multi-class feedback classification based on the Hattie and Timperley model.

Keywords: Feedback, Deep Learning, Multiclass Classification

References

André, M., Mello, R. F., Nascimento, A., Lins, R. D., and Gašević, D. (2021). Toward automatic classification of online discussion messages for social presence. IEEE Transactions on Learning Technologies, 14(6):802–816.

Bagla, K., Kumar, A., Gupta, S., and Gupta, A. (2021). Noisy text data: Achilles’ heel of popular transformer based nlp models. arXiv preprint arXiv:2110.03353.

Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovic, V., and Gasevic, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 605–614.

Boud, D. and Falchikov, N. (2007). Rethinking assessment in higher education: Learning for the longer term. Routledge.

Brookhart, S. M. (2017). How to give effective feedback to your students. ASCD.

Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gasevic, D., and Mello, R. F. (2021a). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2:100027.

Cavalcanti, A. P., de Mello, R. F. L., de Miranda, P. B. C., and de Freitas, F. L. G. (2020a). Análise automática de feedback em ambientes de aprendizagem online. In Anais do XXXI Simpósio Brasileiro de Informatica na Educação, pages 892–901. SBC.

Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento, A., Freitas, F., and Gasevic, D. (2020b). How good is my feedback? a content analysis of written feedback. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge - LAK. ACM

Cavalcanti, A. P., Mello, R. F., Miranda, P., Nascimento, A., and Freitas, F. (2021b). Utilização de recursos linguísticos para classificação automática de mensagens de feedback. In Anais do XXXII Simposio Brasileiro de Informática na Educação, pages 861–872. SBC.

Coates, H., James, R., and Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management, 11:19–36.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

Graves, A. and Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. IJCNN 2005.

Hartmann, N. S., Fonseca, E. R., Shulby, C. D., Treviso, M. V., Rodrigues, J. S., and Aluísio, S. M. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. In Anais do XI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 122–131, Porto Alegre, RS, Brasil. SBC.

Hattie, J. and Gan, M. (2011). Instruction based on feedback. In Handbook of research on learning and instruction, pages 263–285. Routledge.

Hattie, J. and Timperley, H. (2007). The power of feedback. Review of educational research, 77(1):81–112.

Henderson, M., Ajjawi, R., Boud, D., and Molloy, E., editors (2019). The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners. Springer International Publishing, Cham, Switzerland. Google-Books-ID: WyxQxgEACAAJ.

Hossin, M. and Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1.

Joulani, P., Gyorgy, A., and Szepesvari, C. (2013). Online learning under delayed feedback. In International Conference on Machine Learning, pages 1453–1461.

Junior, O. O. B. and Fileto, R. (2021). Investigando coerência em postagens de um fórum de dúvidas em ambiente virtual de aprendizagem com o bert. In Anais do XXXII Simpósio Brasileiro de Informática na Educação, pages 749–759. SBC.

Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., and Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4):150.

Laurillard, D. (1993). Rethinking University Teaching: Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. Routledge.

Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2):199– 218.

Osakwe, I., Chen, G., Whitelock-Wainwright, A., Gasevic, D., Cavalcanti, A. P., and Mello, R. F. (2022). Towards automated content analysis of educational feedback: A multi-language study. Computers and Education: Artificial Intelligence, 3:100059.

Ruiz Alonso, D., Zepeda Cortes, C., Castillo Zacatelco, H., and Carballido Carranza, J. L. (2022). Hyperparameter tuning for multi-label classification of feedbacks in online courses. Journal of Intelligent & Fuzzy Systems, (Preprint):1–9.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252.

Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional science, 18(2):119–144.

Souza, F., Nogueira, R., and Lotufo, R. (2020). BERTimbau: pretrained BERT models for Brazilian Portuguese. In 9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil, October 20-23 (to appear).

Wagner Filho, J. A., Wilkens, R., Idiart, M., and Villavicencio, A. (2018). The brWaC corpus: A new open resource for Brazilian Portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Wang, S., Zhou, W., and Jiang, C. (2020). A survey of word embeddings based on deep learning. Computing, 102(3):717–740.

Xu, B., Wang, N., Chen, T., and Li, M. (2015). Empirical evaluation of rectified activations in convolutional network.

Young, T., Hazarika, D., Poria, S., and Cambria, E. (2018). Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine, 13(3):55–75.

Ypsilandis, G. (2002). Feedback in distance education. Computer Assisted Language Learning, 15(2):167–181.