I know what you coded last summer
Resumo
The outbreak of the COVID-19 pandemic caused a surge in enrollments in online courses. Consequently, this boost in numbers of students affected teachers ability to evaluate exercises and resolve doubts. In this context, tools designed to evaluate and provide feedback on code solutions can be used in programming courses to reduce teachers workload. Nonetheless, even with using such tools, the literature shows that learning how to program is a challenging task. Programming is complex and the programming language employed can also affect students outcomes. Thus, designing good exercises can reduce students difficulties in identifying the problem and help reduce syntax challenges. This research employs learning analytics processes on automatic evaluation tools interaction logs and code solutions to find metrics capable of identifying problematic exercises and their difficulty. In this context, an exercise is considered problematic if students have problems interpreting its description or its solution requires complex programming structures like loops, conditionals and recursion. The data comes from online introductory programming courses. Results show that the computed metrics can identify problematic exercises, as well as those that are being challenging.
Palavras-chave:
Programming, Learning Analytics, Automatic Assessment, Exercise difficulty, Educational Datamining
Referências
Akçapinar, G., Altun, A., and Așkar, P. (2019). Using learning analytics to develop early-warning system for at-risk students. International Journal of Educational Technology in Higher Education, 16(1):1–20.
Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2):83–102.
Aleem, A. and Gore, M. M. (2020). Educational data mining methods: A survey. 2020 IEEE 9th International Conference on Communication Systems and Network Technologies (CSNT), pages 182–188
Arifi, S. M., Abdellah, I. N., Zahi, A., and Benabbou, R. (2015). Automatic program assessment using static and dynamic analysis. In 2015 Third World Conference on Complex Systems (WCCS), pages 1–6.
Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., and Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4):561–599.
Bosse, Y. and Gerosa, M. A. (2015). Reprovações e trancamentos nas disciplinas de introdução programação da universidade de são paulo: Um estudo preliminar. In: Anais do XXIII Workshop sobre Educação em Computação, pages 426–435. SBC.
Caspersen, M. E. (2007). Educating Novices in The Skills of Programming. PhD thesis, Department of Computer Science.
Chaturvedi, M. (2017). Data mining and it’s application in edm domain. In 2017 International Conference on Intelligent Computing and Control Systems (ICICCS), pages 829–834.
Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C. (2009). Introduction to Algorithms, Third Edition. The MIT Press, 3rd edition.
Du Boulay, B. (1986). Some difficulties of learning to program. Journal of Educational Computing Research, 2(1):57–73.
Effenberger, T., Čechák, J., and Pelánek, R. (2019). Measuring difficulty of introductory programming tasks. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale, L@S ’19, New York, NY, USA. Association for Computing Machinery.
Félix, I., Souza, L., Brandão, L., Ferreira, B., and Brandão, A. (2019). ivprog: Programação interativa visual e textual na internet. In: Anais dos Workshops do Congresso Brasileiro de Informatica na Educação, volume 8, page 1164.
Félix, I. M., Ambrósio, A. P., Neves, P. S., Siqueira, J., and Brancher, J. D. (2017). Moodle predicta: A data mining tool for student follow up. In CSEDU (1), pages 339–346.
Gomes, A. and Mendes, A. (2007). Learning to program - difficulties and solutions. In International Conference on Engineering Education, pages 283–287.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, page 86–93, New York, NY, USA. Association for Computing Machinery.
Impey, C. and Formanek, M. (2021). Moocs and 100 days of covid: Enrollment surges in massive open online astronomy classes during the coronavirus pandemic. Social Sciences Humanities Open, 4(1):100177.
Lahtinen, E., Ala-Mutka, K., and Järvinen, H.-M. (2005). A study of the difficulties of novice programmers. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE ’05, pages 14–18, New York, NY, USA. ACM.
Liang, C., Majumdar, R., and Ogata, H. (2021). Learning log-based automatic group formation: system design and classroom implementation study. Research and Practice in Technology Enhanced Learning, 16(1):1–22.
Lima, M., Carvalho, L., Oliveira, E., Oliveira, D., and Pereira, F. (2020). Classificação de dificuldade de questoes de programação com base em métricas de código. In: Anais do XXXI Simposio Brasileiro de Informática na Educação, pages 1323–1332, Porto Alegre, RS, Brasil. SBC
Milne, I. and Rowe, G. (2002). Difficulties in learning and teaching programming—views of students and tutors. Education and Information Technologies, 7(1):55–66.
Papert, S. (1980). Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, Inc., New York, NY, USA.
Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., Devlin, M., and Paterson, J. (2007). A survey of literature on the teaching of introductory programming. SIGCSE Bull., 39(4):204–223.
Pelánek, R., Effenberger, T., and Čechák, J. (2021). Complexity and difficulty of items in learning systems. International Journal of Artificial Intelligence in Education, pages 1–37
Rodríguez del Pino, J. C., Rubio Royo, E., and Hernández Figueroa, Z. J. (2010). Vpl: laboratorio virtual de programacíon para moodle. In: XVI Jornadas de Enseñanza Universitaria de la Informática, pages 429–435. Universidade de Santiago de Compostela. Escola Técnica Superior d’Enxeñaría.
Siemens, G. and Gasevic, D. (2012). Guest editorial-learning and knowledge analytics. Journal of Educational Technology & Society, 15(3):1–2
Sleeman, D. (1986). The challenges of teaching computer programming. Communications of the ACM, 29(9):840–841.
Teodoro, L. and Kappel, M. A. (2020). Aplicação de técnicas de aprendizado de máquina para predição de risco de evasão escolar em instituições públicas de ensino superior no Brasil. Revista Brasileira de Informática na Educação, 28(0):838–863.
Wang, F. L. and Wong, T.-L. (2008). Designing programming exercises with computer assisted instruction. In International Conference on Hybrid Learning and Education, pages 283–293. Springer.
Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2):83–102.
Aleem, A. and Gore, M. M. (2020). Educational data mining methods: A survey. 2020 IEEE 9th International Conference on Communication Systems and Network Technologies (CSNT), pages 182–188
Arifi, S. M., Abdellah, I. N., Zahi, A., and Benabbou, R. (2015). Automatic program assessment using static and dynamic analysis. In 2015 Third World Conference on Complex Systems (WCCS), pages 1–6.
Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., and Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4):561–599.
Bosse, Y. and Gerosa, M. A. (2015). Reprovações e trancamentos nas disciplinas de introdução programação da universidade de são paulo: Um estudo preliminar. In: Anais do XXIII Workshop sobre Educação em Computação, pages 426–435. SBC.
Caspersen, M. E. (2007). Educating Novices in The Skills of Programming. PhD thesis, Department of Computer Science.
Chaturvedi, M. (2017). Data mining and it’s application in edm domain. In 2017 International Conference on Intelligent Computing and Control Systems (ICICCS), pages 829–834.
Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C. (2009). Introduction to Algorithms, Third Edition. The MIT Press, 3rd edition.
Du Boulay, B. (1986). Some difficulties of learning to program. Journal of Educational Computing Research, 2(1):57–73.
Effenberger, T., Čechák, J., and Pelánek, R. (2019). Measuring difficulty of introductory programming tasks. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale, L@S ’19, New York, NY, USA. Association for Computing Machinery.
Félix, I., Souza, L., Brandão, L., Ferreira, B., and Brandão, A. (2019). ivprog: Programação interativa visual e textual na internet. In: Anais dos Workshops do Congresso Brasileiro de Informatica na Educação, volume 8, page 1164.
Félix, I. M., Ambrósio, A. P., Neves, P. S., Siqueira, J., and Brancher, J. D. (2017). Moodle predicta: A data mining tool for student follow up. In CSEDU (1), pages 339–346.
Gomes, A. and Mendes, A. (2007). Learning to program - difficulties and solutions. In International Conference on Engineering Education, pages 283–287.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, page 86–93, New York, NY, USA. Association for Computing Machinery.
Impey, C. and Formanek, M. (2021). Moocs and 100 days of covid: Enrollment surges in massive open online astronomy classes during the coronavirus pandemic. Social Sciences Humanities Open, 4(1):100177.
Lahtinen, E., Ala-Mutka, K., and Järvinen, H.-M. (2005). A study of the difficulties of novice programmers. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE ’05, pages 14–18, New York, NY, USA. ACM.
Liang, C., Majumdar, R., and Ogata, H. (2021). Learning log-based automatic group formation: system design and classroom implementation study. Research and Practice in Technology Enhanced Learning, 16(1):1–22.
Lima, M., Carvalho, L., Oliveira, E., Oliveira, D., and Pereira, F. (2020). Classificação de dificuldade de questoes de programação com base em métricas de código. In: Anais do XXXI Simposio Brasileiro de Informática na Educação, pages 1323–1332, Porto Alegre, RS, Brasil. SBC
Milne, I. and Rowe, G. (2002). Difficulties in learning and teaching programming—views of students and tutors. Education and Information Technologies, 7(1):55–66.
Papert, S. (1980). Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, Inc., New York, NY, USA.
Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., Devlin, M., and Paterson, J. (2007). A survey of literature on the teaching of introductory programming. SIGCSE Bull., 39(4):204–223.
Pelánek, R., Effenberger, T., and Čechák, J. (2021). Complexity and difficulty of items in learning systems. International Journal of Artificial Intelligence in Education, pages 1–37
Rodríguez del Pino, J. C., Rubio Royo, E., and Hernández Figueroa, Z. J. (2010). Vpl: laboratorio virtual de programacíon para moodle. In: XVI Jornadas de Enseñanza Universitaria de la Informática, pages 429–435. Universidade de Santiago de Compostela. Escola Técnica Superior d’Enxeñaría.
Siemens, G. and Gasevic, D. (2012). Guest editorial-learning and knowledge analytics. Journal of Educational Technology & Society, 15(3):1–2
Sleeman, D. (1986). The challenges of teaching computer programming. Communications of the ACM, 29(9):840–841.
Teodoro, L. and Kappel, M. A. (2020). Aplicação de técnicas de aprendizado de máquina para predição de risco de evasão escolar em instituições públicas de ensino superior no Brasil. Revista Brasileira de Informática na Educação, 28(0):838–863.
Wang, F. L. and Wong, T.-L. (2008). Designing programming exercises with computer assisted instruction. In International Conference on Hybrid Learning and Education, pages 283–293. Springer.
Publicado
22/11/2021
Como Citar
DE SOUZA, Lucas Mendonça; FELIX, Igor Moreira; FERREIRA, Bernardo Martins; BRANDÃO, Anarosa Alves Franco; BRANDÃO, Leônidas de Oliveira.
I know what you coded last summer. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 32. , 2021, Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 909-920.
DOI: https://doi.org/10.5753/sbie.2021.218673.
