Automatic Problem Recommendation in Online Judges Using Natural Language Processing and Data-Driven Analysis

• Hermino Barbosa de Freitas Júnior Universidade Federal de Roraima
• Filipe Dwan Pereira Federal University of Roraima https://orcid.org/0000-0003-4914-3347
• Elaine Harada Teixeira de Oliveira Federal University of Amazonas https://orcid.org/0000-0003-2884-9359
• David Braga Fernandes de Oliveira Federal University of Amazonas
• Leandro Silva Galvão de Carvalho Federal University of Amazonas https://orcid.org/0000-0003-2970-2084

DOI: https://doi.org/10.5753/cbie.sbie.2020.1152

Abstract

Typically, learners struggle to find suitable problems in online judges due to the huge volume of problems available on these systems. In this sense, we propose and validate methods for automatic recommendation of problems in online judges, where the recommendations are made based on a target problem, previously solved by the learner. For validation of our proposed methods, we used as a baseline a stochastic method that randomly recommends questions chosen from assignments created by instructors. In total, 15 students and 3 instructors evaluated 324 problems recommended by our methods, using a double blind control approach. As a result, we showed that our methods presented better recommendations for the students in terms of effort employed and success achieved (higher success rate and lower failure and dropout rate). In closing, we believe that our methods can be used to support instructors of selecting problems to create assignment lists.

Keywords: Online Judges, Recommendation Systems, Data Driven Analysis, Natural Language Processing, Programming Challenges

References

Aljohani, T., Pereira, F. D., Cristea, A. I., e Oliveira, E. (2020). Prediction of users’ professional profile in moocs only by utilising learners’ written texts. In International Conference on Intelligent Tutoring Systems, pages 163–173. Springer.


Bradley, S. (2020). Creative assessment in programming: Diversity and divergence. In Proceedings of the 4th Conference on Computing Education Practice 2020, pages 1–4.


Chau, H., Barria-Pineda, J., e Brusilovsky, P. (2017). Content wizard: concept-based recommender system for instructors of programming courses. In Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, pages 135–140. ACM.


Dwan, F., Oliveira, E., e Fernandes, D. (2017). Predição de zona de aprendizagem de alunos de introdução a programação em ambientes de correção automática de código. In Simposio Brasileiro de Informática na Educação (SBIE) , volume 28, page 1507.


Fonseca, S., Pereira, F. D., Oliveira, E. H. T., Olveira, D. B. F., Carvalho, L. G., e Cristea, A. I. (2020). Automatic subject-based contextualisation of programming assignment lists. In Proc. of 13th International Conference on Educational Data Mining, pages 81–91. EDM.


Hosseini, R. e Brusilovsky, P. (2017). A study of concept-based similarity approaches for recommending program examples. New Review of Hypermedia and Multimedia, 23(3):161–188.


Paula, L. C., Oliveira Fassbinder, A. G., e Barbosa, E. F. (2014). A recommendation system to support the students performance in programming contests. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pages 1–8. IEEE.


Pereira, F. D., Oliveira, E. H., Fernandes, D., e Cristea, A. (2019). Early performance prediction for cs1 course students using a combination of machine learning and an evolutionary algorithm. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), volume 2161, pages 183–184. IEEE.


Pereira, F. D., Oliveira, E. H. T., Oliveira, D., Cristea, A. I., Carvalho, L., Fonseca, S., Toda, A., e Isotani, S. (2020). Using learning analytics in the Amazonas: understanding students’ behaviour in CS1. British journal of educational technology.


Pereira, F. D., Oliveira, E. H. T., e Oliveira, D. F. B. (2018). Uso de um método preditivo para inferir a zona de aprendizagem de alunos de programação em um ambiente de correção automática de código. Mestrado em informática, Universidade Federal do Amazonas, Manaus.


Yera, R. e Martínez, L. (2017). A recommendation approach for programming online judges supported by data preprocessing techniques. Applied Intelligence, 47(2):277–290.


Zhao, W. X., Zhang, W., He, Y., Xie, X., e Wen, J.-R. (2018). Automatically learning topics and difficulty levels of problems in online judge systems. ACM Transactions on Information Systems (TOIS), 36(3):1–33.