Classificação Multi-classe para Análise de Qualidade de Feedback
Resumo
O feedback é um fator muito importante no processo de ensino-aprendizagem e crucial na Educação a Distância, pois, como professores e alunos estão separados no espaço e/ou tempo, é através do feedback que o aluno vai entender como está o seu desempenho na disciplina e quais são os próximos passos do aprendizado. Existem na literatura modelos de feedback que ajudam o professor a estruturar e fornecer um feedback de qualidade ao aluno. Nesse trabalho utilizamos o conceituado modelo de feedback de Hattie e Timperley que divide o feedback em categorias (tarefa, processamento da tarefa, regulação e pessoal). É possível encontrar na literatura trabalhos que analisam feedback automaticamente com base nesse modelo. Contudo, esses trabalhos utilizam algoritmos tradicionais de aprendizagem de máquina e treinam classificadores binários para cada nível de feedback. Dessa forma, este trabalho tem como objetivo utilizar algoritmos de deep learning para classificação multi-classe de feedback com base no modelo de Hattie e Timperley.
Palavras-chave:
Feedback, Aprendizado Profundo, Classificação Multiclasse
Referências
André, M., Mello, R. F., Nascimento, A., Lins, R. D., and Gašević, D. (2021). Toward automatic classification of online discussion messages for social presence. IEEE Transactions on Learning Technologies, 14(6):802–816.
Bagla, K., Kumar, A., Gupta, S., and Gupta, A. (2021). Noisy text data: Achilles’ heel of popular transformer based nlp models. arXiv preprint arXiv:2110.03353.
Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovic, V., and Gasevic, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 605–614.
Boud, D. and Falchikov, N. (2007). Rethinking assessment in higher education: Learning for the longer term. Routledge.
Brookhart, S. M. (2017). How to give effective feedback to your students. ASCD.
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gasevic, D., and Mello, R. F. (2021a). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2:100027.
Cavalcanti, A. P., de Mello, R. F. L., de Miranda, P. B. C., and de Freitas, F. L. G. (2020a). Análise automática de feedback em ambientes de aprendizagem online. In Anais do XXXI Simpósio Brasileiro de Informatica na Educação, pages 892–901. SBC.
Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento, A., Freitas, F., and Gasevic, D. (2020b). How good is my feedback? a content analysis of written feedback. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge - LAK. ACM
Cavalcanti, A. P., Mello, R. F., Miranda, P., Nascimento, A., and Freitas, F. (2021b). Utilização de recursos linguísticos para classificação automática de mensagens de feedback. In Anais do XXXII Simposio Brasileiro de Informática na Educação, pages 861–872. SBC.
Coates, H., James, R., and Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management, 11:19–36.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
Graves, A. and Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. IJCNN 2005.
Hartmann, N. S., Fonseca, E. R., Shulby, C. D., Treviso, M. V., Rodrigues, J. S., and Aluísio, S. M. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. In Anais do XI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 122–131, Porto Alegre, RS, Brasil. SBC.
Hattie, J. and Gan, M. (2011). Instruction based on feedback. In Handbook of research on learning and instruction, pages 263–285. Routledge.
Hattie, J. and Timperley, H. (2007). The power of feedback. Review of educational research, 77(1):81–112.
Henderson, M., Ajjawi, R., Boud, D., and Molloy, E., editors (2019). The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners. Springer International Publishing, Cham, Switzerland. Google-Books-ID: WyxQxgEACAAJ.
Hossin, M. and Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1.
Joulani, P., Gyorgy, A., and Szepesvari, C. (2013). Online learning under delayed feedback. In International Conference on Machine Learning, pages 1453–1461.
Junior, O. O. B. and Fileto, R. (2021). Investigando coerência em postagens de um fórum de dúvidas em ambiente virtual de aprendizagem com o bert. In Anais do XXXII Simpósio Brasileiro de Informática na Educação, pages 749–759. SBC.
Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., and Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4):150.
Laurillard, D. (1993). Rethinking University Teaching: Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. Routledge.
Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2):199– 218.
Osakwe, I., Chen, G., Whitelock-Wainwright, A., Gasevic, D., Cavalcanti, A. P., and Mello, R. F. (2022). Towards automated content analysis of educational feedback: A multi-language study. Computers and Education: Artificial Intelligence, 3:100059.
Ruiz Alonso, D., Zepeda Cortes, C., Castillo Zacatelco, H., and Carballido Carranza, J. L. (2022). Hyperparameter tuning for multi-label classification of feedbacks in online courses. Journal of Intelligent & Fuzzy Systems, (Preprint):1–9.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252.
Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional science, 18(2):119–144.
Souza, F., Nogueira, R., and Lotufo, R. (2020). BERTimbau: pretrained BERT models for Brazilian Portuguese. In 9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil, October 20-23 (to appear).
Wagner Filho, J. A., Wilkens, R., Idiart, M., and Villavicencio, A. (2018). The brWaC corpus: A new open resource for Brazilian Portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Wang, S., Zhou, W., and Jiang, C. (2020). A survey of word embeddings based on deep learning. Computing, 102(3):717–740.
Xu, B., Wang, N., Chen, T., and Li, M. (2015). Empirical evaluation of rectified activations in convolutional network.
Young, T., Hazarika, D., Poria, S., and Cambria, E. (2018). Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine, 13(3):55–75.
Ypsilandis, G. (2002). Feedback in distance education. Computer Assisted Language Learning, 15(2):167–181.
Bagla, K., Kumar, A., Gupta, S., and Gupta, A. (2021). Noisy text data: Achilles’ heel of popular transformer based nlp models. arXiv preprint arXiv:2110.03353.
Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovic, V., and Gasevic, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 605–614.
Boud, D. and Falchikov, N. (2007). Rethinking assessment in higher education: Learning for the longer term. Routledge.
Brookhart, S. M. (2017). How to give effective feedback to your students. ASCD.
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gasevic, D., and Mello, R. F. (2021a). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2:100027.
Cavalcanti, A. P., de Mello, R. F. L., de Miranda, P. B. C., and de Freitas, F. L. G. (2020a). Análise automática de feedback em ambientes de aprendizagem online. In Anais do XXXI Simpósio Brasileiro de Informatica na Educação, pages 892–901. SBC.
Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento, A., Freitas, F., and Gasevic, D. (2020b). How good is my feedback? a content analysis of written feedback. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge - LAK. ACM
Cavalcanti, A. P., Mello, R. F., Miranda, P., Nascimento, A., and Freitas, F. (2021b). Utilização de recursos linguísticos para classificação automática de mensagens de feedback. In Anais do XXXII Simposio Brasileiro de Informática na Educação, pages 861–872. SBC.
Coates, H., James, R., and Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management, 11:19–36.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
Graves, A. and Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610. IJCNN 2005.
Hartmann, N. S., Fonseca, E. R., Shulby, C. D., Treviso, M. V., Rodrigues, J. S., and Aluísio, S. M. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. In Anais do XI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 122–131, Porto Alegre, RS, Brasil. SBC.
Hattie, J. and Gan, M. (2011). Instruction based on feedback. In Handbook of research on learning and instruction, pages 263–285. Routledge.
Hattie, J. and Timperley, H. (2007). The power of feedback. Review of educational research, 77(1):81–112.
Henderson, M., Ajjawi, R., Boud, D., and Molloy, E., editors (2019). The Impact of Feedback in Higher Education: Improving Assessment Outcomes for Learners. Springer International Publishing, Cham, Switzerland. Google-Books-ID: WyxQxgEACAAJ.
Hossin, M. and Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1.
Joulani, P., Gyorgy, A., and Szepesvari, C. (2013). Online learning under delayed feedback. In International Conference on Machine Learning, pages 1453–1461.
Junior, O. O. B. and Fileto, R. (2021). Investigando coerência em postagens de um fórum de dúvidas em ambiente virtual de aprendizagem com o bert. In Anais do XXXII Simpósio Brasileiro de Informática na Educação, pages 749–759. SBC.
Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., and Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4):150.
Laurillard, D. (1993). Rethinking University Teaching: Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. Routledge.
Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2):199– 218.
Osakwe, I., Chen, G., Whitelock-Wainwright, A., Gasevic, D., Cavalcanti, A. P., and Mello, R. F. (2022). Towards automated content analysis of educational feedback: A multi-language study. Computers and Education: Artificial Intelligence, 3:100059.
Ruiz Alonso, D., Zepeda Cortes, C., Castillo Zacatelco, H., and Carballido Carranza, J. L. (2022). Hyperparameter tuning for multi-label classification of feedbacks in online courses. Journal of Intelligent & Fuzzy Systems, (Preprint):1–9.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252.
Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional science, 18(2):119–144.
Souza, F., Nogueira, R., and Lotufo, R. (2020). BERTimbau: pretrained BERT models for Brazilian Portuguese. In 9th Brazilian Conference on Intelligent Systems, BRACIS, Rio Grande do Sul, Brazil, October 20-23 (to appear).
Wagner Filho, J. A., Wilkens, R., Idiart, M., and Villavicencio, A. (2018). The brWaC corpus: A new open resource for Brazilian Portuguese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Wang, S., Zhou, W., and Jiang, C. (2020). A survey of word embeddings based on deep learning. Computing, 102(3):717–740.
Xu, B., Wang, N., Chen, T., and Li, M. (2015). Empirical evaluation of rectified activations in convolutional network.
Young, T., Hazarika, D., Poria, S., and Cambria, E. (2018). Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine, 13(3):55–75.
Ypsilandis, G. (2002). Feedback in distance education. Computer Assisted Language Learning, 15(2):167–181.
Publicado
16/11/2022
Como Citar
BATISTA, Hyan H. N.; CAVALCANTI, Anderson Pinheiro; MIRANDA, Péricles; NASCIMENTO, André; FERREIRA MELLO, Rafael.
Classificação Multi-classe para Análise de Qualidade de Feedback. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 33. , 2022, Manaus.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 1114-1125.
DOI: https://doi.org/10.5753/sbie.2022.225396.
