Transition Times in Correct and Error States as Performance Indicators in Online Judges

• Ingrid Lima dos Santos Federal University of Amazonas
• David Braga Fernandes Oliveira Federal University of Amazonas
• Leandro Silva Galvão de Carvalho Federal University of Amazonas
• Filipe Dwan Pereira Federal University of Roraima
• Elaine Harada Teixeira de Oliveira Federal University of Amazonas https://orcid.org/0000-0003-2884-9359

DOI: https://doi.org/10.5753/cbie.sbie.2020.1283

Abstract

Introduction to Programming (CS1) classes typically show high failure rates. Indeed, many studies have been carried out to understand how students develop code solutions and which behaviors may impact positively or negatively on their performance. In this sense, this study proposes and validates a model of state transitions in an Online Judge (OJ) with an emphasis on CS1 classes. The model takes into account the transition time of error and correctness states of the solutions developed by 489 students from 9 CS1 classes in an OJ. As a result, we could find a moderate correlation between the time spent on incorrect and correct transitions, and the CS1 student final grade.

Keywords: online judge, state model, performance indicators

References

Alves, A., de Carvalho, L. S. G., Oliveira, E., and Fernandes, D. (2019). Análise comportamental em juízes online para predição do desempenho final de alunos em disciplinas de computação. In Simpósio Brasileiro de Informática na Educação (SBIE), volume 30, page 1906.


Baker, R. S. and Inventado, P. S. (2014). Educational data mining and learning analytics.In Learning analytics, pages 61–75. Springer.


Carter, A. S., Hundhausen, C. D., and Adesope, O. (2015). The normalized programmingstate model: Predicting student performance in computing courses based on programming behavior. In Proceedings of the eleventh annual International Conference onInternational Computing Education Research, pages 141–150. ACM.


Dwan, F., Oliveira, E., and Fernandes, D. (2017). Predição de zona de aprendizagem de alunos de introdução à programação em ambientes de correção automática de código. In Simpósio Brasileiro de Informática na Educação (SBIE), volume 28, page 1507.


Fonseca, S., Oliveira, E., Pereira, F., Fernandes, D., and de Carvalho, L. S. G.(2019). Adaptação de um método preditivo para inferir o desempenho de alunos de programação. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 1651.


Hauke, J. and Kossowski, T. (2011). Comparison of values of pearson’s and spearman’s correlation coefficients on the same sets of data. Quaestiones geographicae, 30(2):87–93.


Jadud, M. C. (2006). Methods and tools for exploring novice compilation behaviour. In Proceedings of the second international workshop on Computing education research,pages 73–84. ACM.


Pereira, F., Oliveira, E., Fernandes, D., de Carvalho, L. S. G., and Junior, H. (2019).Otimização e automação da predição precoce do desempenho de alunos que utilizam juízes online: uma abordagem com algoritmo genético. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 1451.


Pereira, F. D., Oliveira, E. H., Oliveira, D. B., Cristea, A. I., Carvalho, L. S., Fonseca,S. C., Toda, A., and Isotani, S. (2020). Using learning analytics in the amazonas: understanding students’ behaviour in introductory programming. British Journal of Educational Technology.


Pereira, F. D., Oliveira, E. H. T., and Oliveira, D. F. B. (2018). Uso de um método preditivo para inferir a zona de aprendizagem de alunos de programação em um ambiente de correção automática de código. Mestrado em informática, Universidade Federal do Amazonas, Manaus.


Robins, A. V. (2019). Novice programmers and introductory programming. In The Cambridge Handbook of Computing Education Research, chapter 12, pages 327–376. Cambridge University Pres, Cambridge.


Tabanao, E. S., Rodrigo, M. M. T., and Jadud, M. C. (2011). Predicting at-risk novice java programmers through the analysis of online protocols. In Proceedings of the seventh international workshop on Computing education research, pages 85–92. ACM.


Wasik, S., Antczak, M., Badura, J., Laskowski, A., and Sternal, T. (2018). A survey on online judge systems and their applications. ACM Computing Surveys (CSUR),51(1):1–34.


Watson, C. and Li, F. W. (2014). Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation & technology in computer science education, pages 39–44.