Recommendation of Problems in Online Judges Using Natural Language Processing Techniques and Data-Driven Analysis
Abstract
Typically, learners struggle to find suitable problems in online judges due to the huge volume of problems available on these systems. In this sense, we propose and validate methods for automatic recommendation of problems in online judges, where the recommendations are made based on a target problem, previously solved by the learner. The first proposed method uses natural language processing over the problems' statements to make the recommendations. The second method uses the data-driven behavior of the programming students on an online judge called CodeBench. For validation of our proposed methods, we used as a baseline a stochastic method that randomly recommends questions chosen from assignments created by instructors. In total, 15 students and 3 instructors evaluated 324 problems recommended by our methods, using a double blind control approach. As a result, we showed that our methods presented better recommendations for the students in terms of effort employed and success achieved (higher success rate and lower failure and dropout rate). In closing, we believe that our methods can be used to support instructors of selecting problems to create assignment lists.
