Automatic creating variation of CS1 assignments and exams
Resumo
The adoption of Online Judge (OJ) environments by CS1 instructors has increased over the last few years. A potential explanation of this growth is that OJs reduce instructors? workload in correcting learners? codes and provide instantaneous and accurate feedback to students about the correctness of their solutions. Despite the benefits, there are still repetitive and laborious tasks to feed OJ systems. For example, the literature recommends that instructors create variations of assignments and exams for different CS1 classes during the semesters to hamper plagiarism practice. By creating variations of assignments and exams, it is more difficult for students to use code solutions from past courses. To address this, we intend to create a mechanism for automatically selecting problems to compose new assignments and exams so that the new selection of problems is similar enough to the old in terms of problem topics and challenge levels. A video explaining this work can be found at the following link: https://youtu.be/ANqMD-ZAURk.
Palavras-chave:
recommendation system, online judges, CS1, data-driven analysis
Referências
Ibrahim Albluwi. 2019. Plagiarism in programming assessments: a systematic review. ACM Transactions on Computing Education (TOCE) 20, 1 (2019), 1–28.
Tahani Aljohani, Filipe Dwan Pereira, Alexandra I Cristea, and Elaine Oliveira. 2020. Prediction of Users’ Professional Profile in MOOCs Only by Utilising Learners’ Written Texts. In International Conference on Intelligent Tutoring Systems. Springer, 163–173.
Joe Michael Allen and Frank Vahid. 2021. Concise Graphical Representations of Student Effort on Weekly Many Small Programs. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 349–354.
Ada Araujo, Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2021. Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 123–131.
Steven Bradley. 2016. Managing plagiarism in programming assignments with blended assessment and randomisation. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research. 21–30.
Hermino Barbosa de Freitas Júnior, Filipe Dwan Pereira, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Recomendação Automática de Problemas em Juízes Online Usando Processamento de Linguagem Natural e Análise Dirigida aos Dados. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1152–1161.
Marcos Avner Pimenta de Lima Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Filipe Dwan Pereira. 2021. Uso de atributos de código para classificação da facilidade de questões de codificação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 113–122.
Joseph de Oliveira, Felipe Salem, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, and Filipe Dwan Pereira. 2020. Os estudantes leem as mensagens de feedback estendido exibidas em juízes online?. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1723–1732.
Aracele Garcia de Oliveira Fassbinder, Tiago Gonçalves Gonçalves Botelho, Ricardo José Martins, and Ellen Francine Barbosa. 2015. Applying flipped classroom and problem-based learning in a CS1 course. In 2015 IEEE Frontiers in Education Conference (FIE). IEEE, 1–7.
Ingrid Lima dos Santos, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and Elaine Harada Teixeira de Oliveira. 2020. Tempos de Transição em Estados de Corretude e Erro como Indicadores de Desempenho em Juízes Online. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1283–1292.
Filipe Dwan, Elaine Oliveira, and David Fernandes. 2017. Predição de zona de aprendizagem de alunos de introdução à programação em ambientes de correção automática de código. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 28. 1507.
Samuel C Fonseca, Filipe Dwan Pereira, Elaine HT Oliveira, David BF Oliveira, Leandro SG Carvalho, and Alexandra I Cristea. 2020. Automatic Subject-based Contextualisation of Programming Assignment Lists. EDM.
Max Fowler and Craig Zilles. 2021. Superficial Code-guise: Investigating the Impact of Surface Feature Changes on Students’ Programming Question Scores. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 3–9.
Rodrigo Elias Francisco and Ana Paula Ambrosio. 2015. Mining an Online Judge System to Support Introductory Computer Programming Teaching.. In EDM (Workshops). Citeseer.
Marcos Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, and Filipe Dwan Pereira. 2020. Classificação de dificuldade de questões de programação com base em métricas de código. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1323–1332.
Andrew Luxton-Reilly, Ibrahim Albluwi, Brett A Becker, Michail Giannakos, Amruth N Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory programming: a systematic literature review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education. 55–106.
Filipe Pereira, Elaine Oliveira, David Fernandes, Hermino Junior, and Leandro Silva Galvão de Carvalho. 2019. Otimização e automação da predição precoce do desempenho de alunos que utilizam juízes online: uma abordagem com algoritmo genético. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 30. 1451.
Filipe Dwan Pereira, Linnik Maciel de Souza, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Predição de desempenho em ambientes computacionais para turmas de programação: um Mapeamento Sistemático da Literatura. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1673–1682.
Filipe Dwan Pereira, Samuel C Fonseca, Elaine HT Oliveira, David BF Oliveira, Alexandra I Cristea, and Leandro SG Carvalho. 2020. Deep learning for early performance prediction of introductory programming students: a comparative and explanatory study. Brazilian journal of computers in education. 28 (2020), 723–749.
Filipe D Pereira, Hermino Junior, Luiz Rodriguez, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, David Oliveira, Leandro Carvalho, Samuel Fonseca, Ahmed Alamri, and Seiji Isotani. 2021. A recommender system based on effort: towards minimising negative affects and maximising achievement in CS1 learning. In International Conference on Intelligent Tutoring Systems. Springer.
Filipe D Pereira, Elaine Oliveira, Alexandra Cristea, David Fernandes, Luciano Silva, Gene Aguiar, Ahmed Alamri, and Mohammad Alshehri. 2019. Early dropout prediction for programming courses supported by online judges. In International Conference on Artificial Intelligence in Education. Springer, 67–72.
Filipe Dwan Pereira, Elaine HT Oliveira, David Fernandes, and Alexandra Cristea. 2019. Early performance prediction for CS1 course students using a combination of machine learning and an evolutionary algorithm. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Vol. 2161. IEEE, 183–184.
Filipe Dwan Pereira, Elaine HT Oliveira, David Oliveira, Alexandra I Cristea, Leandro Carvalho, Samuel Fonseca, Armando Toda, and Seiji Isotani. 2020. Using learning analytics in the Amazonas: understanding students’ behaviour in introductory programming. British journal of educational technology. (2020).
Filipe Dwan Pereira, Elaine H T Oliveira, and David F B Oliveira. 2018. Uso de um método preditivo para inferir a zona de aprendizagem de alunos de programação em um ambiente de correção automática de código. Mestrado em Informática. Universidade Federal do Amazonas, Manaus.
Filipe Dwan Pereira, Francisco Pires, Samuel C Fonseca, Elaine HT Oliveira, Leandro SG Carvalho, David BF Oliveira, and Alexandra I. 2021. Towards a Human-AI hybrid system for categorising programming problems (SIGCSE ’21). Association for Computing Machinery, New York, NY, USA, 7 pages.
Filipe D Pereira, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, Seiji Isotani, Dion Laranjeira, Adriano Almeida, and Jonas Mendonça. 2020. Can we use gamification to predict students’ performance? A case study supported by an online judge. In International Conference on Intelligent Tutoring Systems. Springer, 259–269.
Tomohiro Saito and Yutaka Watanobe. 2020. Learning Path Recommendation System for Programming Education based on Neural Networks. International Journal of Distance Education Technologies (IJDET) 18, 1 (2020), 36–64.
Szymon Wasik, Maciej Antczak, Jan Badura, Artur Laskowski, and Tomasz Sternal. 2018. A survey on online judge systems and their applications. ACM Computing Surveys (CSUR) 51, 1 (2018), 3.
Ruiguo Yu, Zhiyong Cai, Xiuping Du, Muwen He, Zan Wang, Binlan Yang, and Peng Chang. 2015. The research of the recommendation algorithm in online learning. International Journal of Multimedia and Ubiquitous Engineering 10, 4 (2015), 71–80.
Wayne Xin Zhao, Wenhui Zhang, Yulan He, Xing Xie, and Ji-Rong Wen. 2018. Automatically learning topics and difficulty levels of problems in online judge systems. ACM Transactions on Information Systems (TOIS) 36, 3 (2018), 27.
Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Marcela Pessoa, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2020. Uma análise orientada a dados para avaliar o impacto da gamificação de um juiz on-line no desempenho de estudantes. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 491–500.
Tahani Aljohani, Filipe Dwan Pereira, Alexandra I Cristea, and Elaine Oliveira. 2020. Prediction of Users’ Professional Profile in MOOCs Only by Utilising Learners’ Written Texts. In International Conference on Intelligent Tutoring Systems. Springer, 163–173.
Joe Michael Allen and Frank Vahid. 2021. Concise Graphical Representations of Student Effort on Weekly Many Small Programs. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 349–354.
Ada Araujo, Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2021. Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 123–131.
Steven Bradley. 2016. Managing plagiarism in programming assignments with blended assessment and randomisation. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research. 21–30.
Hermino Barbosa de Freitas Júnior, Filipe Dwan Pereira, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Recomendação Automática de Problemas em Juízes Online Usando Processamento de Linguagem Natural e Análise Dirigida aos Dados. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1152–1161.
Marcos Avner Pimenta de Lima Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Filipe Dwan Pereira. 2021. Uso de atributos de código para classificação da facilidade de questões de codificação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 113–122.
Joseph de Oliveira, Felipe Salem, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, and Filipe Dwan Pereira. 2020. Os estudantes leem as mensagens de feedback estendido exibidas em juízes online?. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1723–1732.
Aracele Garcia de Oliveira Fassbinder, Tiago Gonçalves Gonçalves Botelho, Ricardo José Martins, and Ellen Francine Barbosa. 2015. Applying flipped classroom and problem-based learning in a CS1 course. In 2015 IEEE Frontiers in Education Conference (FIE). IEEE, 1–7.
Ingrid Lima dos Santos, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and Elaine Harada Teixeira de Oliveira. 2020. Tempos de Transição em Estados de Corretude e Erro como Indicadores de Desempenho em Juízes Online. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1283–1292.
Filipe Dwan, Elaine Oliveira, and David Fernandes. 2017. Predição de zona de aprendizagem de alunos de introdução à programação em ambientes de correção automática de código. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 28. 1507.
Samuel C Fonseca, Filipe Dwan Pereira, Elaine HT Oliveira, David BF Oliveira, Leandro SG Carvalho, and Alexandra I Cristea. 2020. Automatic Subject-based Contextualisation of Programming Assignment Lists. EDM.
Max Fowler and Craig Zilles. 2021. Superficial Code-guise: Investigating the Impact of Surface Feature Changes on Students’ Programming Question Scores. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 3–9.
Rodrigo Elias Francisco and Ana Paula Ambrosio. 2015. Mining an Online Judge System to Support Introductory Computer Programming Teaching.. In EDM (Workshops). Citeseer.
Marcos Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, and Filipe Dwan Pereira. 2020. Classificação de dificuldade de questões de programação com base em métricas de código. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1323–1332.
Andrew Luxton-Reilly, Ibrahim Albluwi, Brett A Becker, Michail Giannakos, Amruth N Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. 2018. Introductory programming: a systematic literature review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education. 55–106.
Filipe Pereira, Elaine Oliveira, David Fernandes, Hermino Junior, and Leandro Silva Galvão de Carvalho. 2019. Otimização e automação da predição precoce do desempenho de alunos que utilizam juízes online: uma abordagem com algoritmo genético. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 30. 1451.
Filipe Dwan Pereira, Linnik Maciel de Souza, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Predição de desempenho em ambientes computacionais para turmas de programação: um Mapeamento Sistemático da Literatura. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1673–1682.
Filipe Dwan Pereira, Samuel C Fonseca, Elaine HT Oliveira, David BF Oliveira, Alexandra I Cristea, and Leandro SG Carvalho. 2020. Deep learning for early performance prediction of introductory programming students: a comparative and explanatory study. Brazilian journal of computers in education. 28 (2020), 723–749.
Filipe D Pereira, Hermino Junior, Luiz Rodriguez, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, David Oliveira, Leandro Carvalho, Samuel Fonseca, Ahmed Alamri, and Seiji Isotani. 2021. A recommender system based on effort: towards minimising negative affects and maximising achievement in CS1 learning. In International Conference on Intelligent Tutoring Systems. Springer.
Filipe D Pereira, Elaine Oliveira, Alexandra Cristea, David Fernandes, Luciano Silva, Gene Aguiar, Ahmed Alamri, and Mohammad Alshehri. 2019. Early dropout prediction for programming courses supported by online judges. In International Conference on Artificial Intelligence in Education. Springer, 67–72.
Filipe Dwan Pereira, Elaine HT Oliveira, David Fernandes, and Alexandra Cristea. 2019. Early performance prediction for CS1 course students using a combination of machine learning and an evolutionary algorithm. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Vol. 2161. IEEE, 183–184.
Filipe Dwan Pereira, Elaine HT Oliveira, David Oliveira, Alexandra I Cristea, Leandro Carvalho, Samuel Fonseca, Armando Toda, and Seiji Isotani. 2020. Using learning analytics in the Amazonas: understanding students’ behaviour in introductory programming. British journal of educational technology. (2020).
Filipe Dwan Pereira, Elaine H T Oliveira, and David F B Oliveira. 2018. Uso de um método preditivo para inferir a zona de aprendizagem de alunos de programação em um ambiente de correção automática de código. Mestrado em Informática. Universidade Federal do Amazonas, Manaus.
Filipe Dwan Pereira, Francisco Pires, Samuel C Fonseca, Elaine HT Oliveira, Leandro SG Carvalho, David BF Oliveira, and Alexandra I. 2021. Towards a Human-AI hybrid system for categorising programming problems (SIGCSE ’21). Association for Computing Machinery, New York, NY, USA, 7 pages.
Filipe D Pereira, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, Seiji Isotani, Dion Laranjeira, Adriano Almeida, and Jonas Mendonça. 2020. Can we use gamification to predict students’ performance? A case study supported by an online judge. In International Conference on Intelligent Tutoring Systems. Springer, 259–269.
Tomohiro Saito and Yutaka Watanobe. 2020. Learning Path Recommendation System for Programming Education based on Neural Networks. International Journal of Distance Education Technologies (IJDET) 18, 1 (2020), 36–64.
Szymon Wasik, Maciej Antczak, Jan Badura, Artur Laskowski, and Tomasz Sternal. 2018. A survey on online judge systems and their applications. ACM Computing Surveys (CSUR) 51, 1 (2018), 3.
Ruiguo Yu, Zhiyong Cai, Xiuping Du, Muwen He, Zan Wang, Binlan Yang, and Peng Chang. 2015. The research of the recommendation algorithm in online learning. International Journal of Multimedia and Ubiquitous Engineering 10, 4 (2015), 71–80.
Wayne Xin Zhao, Wenhui Zhang, Yulan He, Xing Xie, and Ji-Rong Wen. 2018. Automatically learning topics and difficulty levels of problems in online judge systems. ACM Transactions on Information Systems (TOIS) 36, 3 (2018), 27.
Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Marcela Pessoa, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2020. Uma análise orientada a dados para avaliar o impacto da gamificação de um juiz on-line no desempenho de estudantes. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 491–500.
Publicado
26/04/2021
Como Citar
PEREIRA, Filipe Dwan; JÚNIOR, Hermino Barbosa de Freitas ; OLIVEIRA, Elaine Harada Teixeira de; CARVALHO, Leandro Silva Galvão de; OLIVEIRA, David Braga Fernandes de; BENEDICT, Aileen; DORODCHI, Mohsen; CRISTEA, Alexandra Ioana.
Automatic creating variation of CS1 assignments and exams. In: LABORATÓRIO DE IDEIAS - SIMPÓSIO BRASILEIRO DE EDUCAÇÃO EM COMPUTAÇÃO (EDUCOMP), 1. , 2021, On-line.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 21-22.
DOI: https://doi.org/10.5753/educomp_estendido.2021.14856.
