Utilização de Recursos Linguísticos para Classificação Automática de Mensagens de Feedback
Resumo
O feedback possui um papel significativo no processo de aprendizagem de um aluno. Permite que os alunos identifiquem os pontos fracos e melhorem a autorregulação. No entanto, estudos mostram que esta é uma área de grande insatisfação no ensino superior. Com o número cada vez maior de participantes em cursos, fornecer feedback eficaz está se tornando uma tarefa cada vez mais desafiadora. Portanto, este artigo explora o uso da análise de conteúdo automatizada para examinar o feedback fornecido por instrutores com base em modelos conceituados da literatura que fornecem boas práticas e classificam os feedbacks em níveis diferentes. Para isso, foram treinados classificadores binários usando o algoritmo XGBoost em conjunto com recursos linguísticos que extraem características das mensagens de feedback. Os resultados indicam desempenho de classificação eficaz tanto para as boas práticas quanto para os níveis de feedback atingindo valores de até 0,89 de acurácia no conjunto de teste.
Palavras-chave:
Feedback, Recursos Linguísticos, Classificação de Texto
Referências
Barbosa, A., Ferreira, M., Ferreira Mello, R., Dueire Lins, R., and Gasevic, D.(2021). The impact of automatic text translation on classification of online discussions for social and cognitive presences. In LAK21: 11th International Learning Analytics and Knowledge Conference, pages 77–87.
Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovic, V., and Gasevic, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 605–614.
Batista, R. L., Salgado, N., and Barreto, R. (2018). Avaliação e feedback automático em educação apoiada por tecnologia: Um mapeamento sistemático da literatura. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 29, page 178.
Butler, D. L. and Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational Research, 65(3):245–281.
Camelo, R., Justino, S., and de Mello, R. F. L. (2020). Coh-metrix pt-br: uma api web de análise textual para a educação. In Anais dos Workshops do IX Congresso Brasileiro de Informática na Educação, pages 179–186. SBC
Cavalcanti, A., Rolim, V., Mello, R., and Freitas, F. (2020). Uma análise entre boas práticas de feedback em ambientes virtuais de aprendizagem. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 962–971, Porto Alegre, RS, Brasil. SBC.
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gasevic, D., and Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2:100027.
Cavalcanti, A. P., de Mello, R. F. L., de Miranda, P. B. C., and de Freitas, F. L. G. (2020). Análise automática de feedback em ambientes de aprendizagem online. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 892–901. SBC.
Cavalcanti, A. P., de Mello, R. F. L., Rolim, V., André, M., Freitas, F., and Gasevic, D. (2019). An analysis of the use of good feedback practices in online learning courses. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), volume 2161, pages 153–157. IEEE.
Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento,A., Freitas, F., and Gasevic, D. (2020). How good is my feedback? a content analysis of written feedback. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge - LAK. ACM.
Chen, T. and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, San Francisco, California, USA. ACM.
Clark, I. (2012). Formative Assessment: Assessment Is for Self-regulated Learning. Educ. Psychol. Rev., 24(2):205–249.
Dweck, C. S. (1999). Self-theories: Their role in motivation, personality, and development. Self-theories: Their role in motivation, personality, and development. Psychology Press, New York, NY, US. Pages: xiii, 195.
Ferguson, P. (2011). Student perceptions of quality feedback in teacher education. Assess Eval High Educ, 36(1): 51–62.
Ferreira, M., Rolim, V., Mello, R. F., Lins, R. D., Chen, G., and Gasevic, D. (2020). Towards automatic content analysis of social presence in transcripts of online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 141–150.
Ferreira-Mello, R., André, M., Pinheiro, A., Costa, E., and Romero, C. (2019). Text mining in education.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(6): e1332.
Friedman, J., Hastie, T., and Tibshirani, R. (2001).The elements of statistical learning, volume 1. Springer series in statistics New York, NY, USA
Hattie, J. and Gan, M. (2011). Instruction based on feedback. In Handbook of research on learning and instruction pages 263–285. Routledge.
Hattie, J. and Timperley, H. (2007). The power of feedback.Review of educational research, 77(1): 81–112.
Laurillard, D. (1993). Rethinking University Teaching: Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. Routledge.
Liu, M., Li, Y., Xu, W., and Liu, L. (2017). Automated Essay Feedback Generation and Its Impact on Revision. IEEE Trans. Learn. Technol., 10(4): 502–513.
Ma, X., Wijewickrema, S., Zhou, S., Zhou, Y., Mhammedi, Z., O’Leary, S., and Bailey, J. (2017). Adversarial Generation of Real-time Feedback with Neural Networks for Simulation-based Training. InProceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 3763–3769, Melbourne, Australia. International Joint Conferences on Artificial Intelligence Organization.
Mani, I. and Zhang, I. (2003). knn approach to unbalanced data distributions: a case study involving information extraction. In Proceedings of Workshop on Learning from Imbalanced Datasets.
McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
Neuendorf, K. A. (2016).The content analysis guidebook. Sage.
Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2):199–218.
Osakwe, I., Chen, G., Whitelock-Wainwright, A., Gasevic, D., Cavalcanti, A. P., and Mello, R. F. (2021). Towards automated content analysis of feedback: A multi-language study. In Proceedings of the 14th International Conference on Educational Data Mining.
Pan, B. (2018). Application of XGBoost algorithm in hourly PM2.5 concentration pre-diction. IOP Conference Series: Earth and Environmental Science, 113:012127. Publisher: IOP Publishing.
Parikh, A., McReelis, K., and Hodges, B. (2001). Student feedback in problem based learning: a survey of 103 final year students across five ontario medical schools. Medical education, 35(7):632–636.
Price, M., Handley, K., Millar, J., and O’Donovan, B. (2010). Feedback: all that effort, but what is the effect? Assess Eval High Educ, 35(3):277–289
Tausczik, Y. R. and Pennebaker, J. W. (2010). The psychological meaning of words: Liwc and computerized text analysis methods. Journal of language and social psychology, 29(1):24–54.
Villalón, J., Kearney, P., Calvo, R. A., and Reimann, P. (2008). Glosser: Enhanced Feedback for Student Writing Tasks. In 2008 Eighth IEEE International Conference on Advanced Learning Technologies, pages 454–458, Santander, Cantabria, Spain. IEEE
Weaver, M. R. (2006). Do students value feedback? Student perceptions of tutors’ written responses. Assess Eval High Educ, 31(3):379–394.
Xiao, Z., Wang, Y., Fu, K., and Wu, F. (2017). Identifying Different Transportation Modes from Trajectory Data Using Tree-Based Ensemble Classifiers. ISPRS International Journal of Geo-Information, 6(2):57. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.
Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovic, V., and Gasevic, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 605–614.
Batista, R. L., Salgado, N., and Barreto, R. (2018). Avaliação e feedback automático em educação apoiada por tecnologia: Um mapeamento sistemático da literatura. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 29, page 178.
Butler, D. L. and Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational Research, 65(3):245–281.
Camelo, R., Justino, S., and de Mello, R. F. L. (2020). Coh-metrix pt-br: uma api web de análise textual para a educação. In Anais dos Workshops do IX Congresso Brasileiro de Informática na Educação, pages 179–186. SBC
Cavalcanti, A., Rolim, V., Mello, R., and Freitas, F. (2020). Uma análise entre boas práticas de feedback em ambientes virtuais de aprendizagem. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 962–971, Porto Alegre, RS, Brasil. SBC.
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gasevic, D., and Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2:100027.
Cavalcanti, A. P., de Mello, R. F. L., de Miranda, P. B. C., and de Freitas, F. L. G. (2020). Análise automática de feedback em ambientes de aprendizagem online. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 892–901. SBC.
Cavalcanti, A. P., de Mello, R. F. L., Rolim, V., André, M., Freitas, F., and Gasevic, D. (2019). An analysis of the use of good feedback practices in online learning courses. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), volume 2161, pages 153–157. IEEE.
Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento,A., Freitas, F., and Gasevic, D. (2020). How good is my feedback? a content analysis of written feedback. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge - LAK. ACM.
Chen, T. and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, San Francisco, California, USA. ACM.
Clark, I. (2012). Formative Assessment: Assessment Is for Self-regulated Learning. Educ. Psychol. Rev., 24(2):205–249.
Dweck, C. S. (1999). Self-theories: Their role in motivation, personality, and development. Self-theories: Their role in motivation, personality, and development. Psychology Press, New York, NY, US. Pages: xiii, 195.
Ferguson, P. (2011). Student perceptions of quality feedback in teacher education. Assess Eval High Educ, 36(1): 51–62.
Ferreira, M., Rolim, V., Mello, R. F., Lins, R. D., Chen, G., and Gasevic, D. (2020). Towards automatic content analysis of social presence in transcripts of online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, pages 141–150.
Ferreira-Mello, R., André, M., Pinheiro, A., Costa, E., and Romero, C. (2019). Text mining in education.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(6): e1332.
Friedman, J., Hastie, T., and Tibshirani, R. (2001).The elements of statistical learning, volume 1. Springer series in statistics New York, NY, USA
Hattie, J. and Gan, M. (2011). Instruction based on feedback. In Handbook of research on learning and instruction pages 263–285. Routledge.
Hattie, J. and Timperley, H. (2007). The power of feedback.Review of educational research, 77(1): 81–112.
Laurillard, D. (1993). Rethinking University Teaching: Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. Routledge.
Liu, M., Li, Y., Xu, W., and Liu, L. (2017). Automated Essay Feedback Generation and Its Impact on Revision. IEEE Trans. Learn. Technol., 10(4): 502–513.
Ma, X., Wijewickrema, S., Zhou, S., Zhou, Y., Mhammedi, Z., O’Leary, S., and Bailey, J. (2017). Adversarial Generation of Real-time Feedback with Neural Networks for Simulation-based Training. InProceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 3763–3769, Melbourne, Australia. International Joint Conferences on Artificial Intelligence Organization.
Mani, I. and Zhang, I. (2003). knn approach to unbalanced data distributions: a case study involving information extraction. In Proceedings of Workshop on Learning from Imbalanced Datasets.
McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
Neuendorf, K. A. (2016).The content analysis guidebook. Sage.
Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2):199–218.
Osakwe, I., Chen, G., Whitelock-Wainwright, A., Gasevic, D., Cavalcanti, A. P., and Mello, R. F. (2021). Towards automated content analysis of feedback: A multi-language study. In Proceedings of the 14th International Conference on Educational Data Mining.
Pan, B. (2018). Application of XGBoost algorithm in hourly PM2.5 concentration pre-diction. IOP Conference Series: Earth and Environmental Science, 113:012127. Publisher: IOP Publishing.
Parikh, A., McReelis, K., and Hodges, B. (2001). Student feedback in problem based learning: a survey of 103 final year students across five ontario medical schools. Medical education, 35(7):632–636.
Price, M., Handley, K., Millar, J., and O’Donovan, B. (2010). Feedback: all that effort, but what is the effect? Assess Eval High Educ, 35(3):277–289
Tausczik, Y. R. and Pennebaker, J. W. (2010). The psychological meaning of words: Liwc and computerized text analysis methods. Journal of language and social psychology, 29(1):24–54.
Villalón, J., Kearney, P., Calvo, R. A., and Reimann, P. (2008). Glosser: Enhanced Feedback for Student Writing Tasks. In 2008 Eighth IEEE International Conference on Advanced Learning Technologies, pages 454–458, Santander, Cantabria, Spain. IEEE
Weaver, M. R. (2006). Do students value feedback? Student perceptions of tutors’ written responses. Assess Eval High Educ, 31(3):379–394.
Xiao, Z., Wang, Y., Fu, K., and Wu, F. (2017). Identifying Different Transportation Modes from Trajectory Data Using Tree-Based Ensemble Classifiers. ISPRS International Journal of Geo-Information, 6(2):57. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute.
Publicado
22/11/2021
Como Citar
CAVALCANTI, Anderson Pinheiro; MELLO, Rafael Ferreira; MIRANDA, Péricles; NASCIMENTO, André; FREITAS, Fred.
Utilização de Recursos Linguísticos para Classificação Automática de Mensagens de Feedback. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 32. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 861-872.
DOI: https://doi.org/10.5753/sbie.2021.218481.
