Difficulty classification of programming questions based on code metrics

• Marcos Lima Federal University of Amazonas
• Leandro Silva Galvão de Carvalho Federal University of Amazonas https://orcid.org/0000-0003-2970-2084
• Elaine Harada Teixeira de Oliveira Federal University of Amazonas https://orcid.org/0000-0003-2884-9359
• David Braga Fernandes Oliveira Federal University of Amazonas
• Filipe Dwan Pereira Federal University of Roraima

DOI: https://doi.org/10.5753/cbie.sbie.2020.1323

Abstract

While learning how to program using online judges, students should be presented with questions at an increasingly difficult level. Besides, randomized exams should be fair. This work presents a method to classify the difficulty of programming questions based on metrics extracted from the instructor’s solution code. This work analyzed 354 questions in Python and used in some Introduction to Programming (CS1) classes taught from 2017 to 2019. It was found that the use of metrics extracted from code can indeed be used to estimate the difficulty of questions, in addition to elucidating some aspects of the difficulty encountered by students.

Keywords: Difficulty of programming questions, assessment, introduction to programming

References

Denny, P., Cukierman, D., and Bhaskar, J. (2015). Measuring the effect of inventing practice exercises on learning in an introductory programming course. InProceedingsof the 15th Koli Calling Conference on Computing Education Research, pages 13–22.


Effenberger, T.,ˇCechák, J., and Pelánek, R. (2019). Measuring difficulty of introductory programming tasks. InProceedings of the Sixth (2019) ACM Conference on Learning@ Scale, L@S ’19, New York, NY, USA. Association for Computing Machinery.


Elnaffar, S. (2016). Using software metrics to predict the difficulty of code writing questions. In 2016 IEEE Global Engineering Education Conference (EDUCON), pages513–518. IEEE.


Francisco, R. E. and Ambrosio, A. P. (2015). Mining an online judge system to support introductory computer programming teaching. In EDM (Workshops).


Halstead, M. H. (1977).Elements of Software Science (Operating and programming systems Series). Elsevier Science Inc., USA.


INEP (2017). Relatório síntese deárea – Ciência da Computação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira.


Llana, L., E., M.-M., and Pareja-Flores (2012). Flop, a free laboratory of programming.In Proceedings of the 12th Koli Calling International Conference on Computing Edu-cation Research.


McCabe, T. J. (1976). A complexity measure.IEEE Transactions on Software Engineering, SE-2(4):308–320.


Meisalo, V., Sutinen, E., and Torvinen, S. (2004). Classification of exercises in a virtual programming course. In: Frontiers in Education Conference, FIE, volume 3.


Pelanek, R. (2016). Applications of the Elo rating system in adaptive educational systems.Computers & Education, 98:169–179.


Pereira, F. D., Oliveira, E. H., Oliveira, D. B., Cristea, A. I., Carvalho, L. S., Fonseca,S. C., Toda, A., and Isotani, S. (2020). Using learning analytics in the amazonas: understanding students’ behaviour in introductory programming.British Journal of educational Technology.


Santos, P., Carvalho, L. S. G., Oliveira, E., and Fernandes, D. (2019). Classificação dedificuldade de questões de programação com base na inteligibilidade do enunciado.In Simpósio Brasileiro de Informática na Educação (SBIE), volume 30, pages 1886–1895.


Sheard, J., Simon, Carbone, A., Chinn, D., Laakso, M.-J., Clear, T., Raadt, M., D’Souza,D., Harland, J., Lister, R., Philpott, A., and Warburton, G. (2011). Exploring program-ming assessment instruments: A classification scheme for examination questions. In 7th Intern. Workshop on Computing Education Research (ICER), Providence, USA.


Simon, Sheard, J., Carbone, A., Chinn, D., Clear, T., Corney, M., D’Souza, D., Fenwick,J., Harland, J., Laakso, M.-J., and Teague, D. (2013). How difficult are exams? a framework for assessing the complexity of introductory programming exams. In 15th Australasian Computing Education Conference (ACE2013).


Wolpert, D. (1992). Stacked generalization.Neural Networks, 5:241–259.


Zaffalon, F., Vargas, A. P., Souza, R. L., Penna, R., Bez, J., Tonin, N., and Costa Botelho,S. S. (2019). Um estudo comparativo entre dois modelos que estimam a habilidade dosestudantes: Elo e teoria de resposta ao item. InSimpósio Brasileiro de Informática naEducação (SBIE), volume 30, pages 459–468.