Interpretable AI to Understand Early Effective and Ineffective Programming Behaviours from CS1 Learners
Resumo
Building predictive models to estimate the learner performance in the beginning of CS1 courses is essential in education to allow early interventions. However, the educational literature notes the lack of studies on early learner behaviours that can be effective or ineffective, that is, programming behaviours that potentially lead to success or failure, respectively. Hence, beyond the prediction, it is crucial to explain what leads the predictive model to make the decisions (e.g., why a given student s is classified as `passed'), which would allow a better understanding of which early programming behaviours are to be encouraged and triggered. In this work in progress, we use a state-of-the-art unified approach to interpret black-box model predictions, which uses SHapley Additive exPlanations (SHAP) method. SHAP method can be used to explain linearly a complex model (e.g. DL or XGboost) in instance level. In our context of CS1 performance prediction, this method gets the predictive model and the features values for a given student as input and the possibility of explanation of which feature values are increasing or decreasing the learner chances of passing as output. That is, using SHAP we can identify early effective and ineffective behaviours in student-level granularity. More than that, using this local explanation as building blocks, we can also extract global data insight and give a summarisation of the model. A video explaining this work can be found at the following link: https://youtu.be/pd6Ma6uInHo
Palavras-chave:
learning analytics, educational data mining, interpretable AI
Referências
Gene Charles Lima Aguiar and Filipe Dwan Pereira. 2018. Uma Abordagem Data-Driven Para Predição Precoce Da Evasão Em Turmas De Programação Que Utilizam Juízes Online. Graduação em Ciência da Computação, Universidade Federal de Roraima, Boa Vista. Universidade Federal de Roraima, Boa Vista.
Ada Araujo, Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2021. Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 123–131.
Jens Bennedsen and Michael E Caspersen. 2019. Failure rates in introductory programming: 12 years later. ACM inroads 10, 2 (2019), 30–36.
Ryan Bockmon, Stephen Cooper, Jonathan Gratch, Jian Zhang, and Mohsen Dorodchi. 2020. Can Students’ Spatial Skills Predict Their Programming Abilities? (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 446–451.
Adam Carter, Christopher Hundhausen, and Daniel Olivares. 2019. Leveraging the IDE for learning analytics. The Cambridge Handbook of Computing Education Research, Sally A. Fincher and Anthony V. Robins (Eds.). Cambridge University Press, Cambridge (2019), 679–706.
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785 794.
Evandro B Costa, Baldoino Fonseca, Marcelo Almeida Santana, Fabrísia Ferreira de Araújo, and Joilson Rego. 2017. Evaluating the effectiveness of educational data mining techniques for early prediction of students’ academic failure in introductory programming courses. Computers in Human Behavior 73 (2017), 247–256.
Hermino Barbosa de Freitas Júnior, Filipe Dwan Pereira, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Recomendação Automática de Problemas em Juízes Online Usando Processamento de Linguagem Natural e Análise Dirigida aos Dados. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1152–1161.
Marcos Avner Pimenta de Lima Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Filipe Dwan Pereira. 2021. Uso de atributos de código para classificação da facilidade de questões de codificação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 113–122.
Joseph de Oliveira, Felipe Salem, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, and Filipe Dwan Pereira. 2020. Os estudantes leem as mensagens de feedback estendido exibidas em juízes online?. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1723–1732.
Ingrid Lima dos Santos, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and Elaine Harada Teixeira de Oliveira. 2020. Tempos de Transição em Estados de Corretude e Erro como Indicadores de Desempenho em Juízes Online. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1283–1292.
Filipe Dwan, Elaine Oliveira, and David Fernandes. 2017. Predição de zona de aprendizagem de alunos de introdução à programação em ambientes de correção automática de código. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 28. 1507.
Samuel Fonseca, Elaine Oliveira, Filipe Pereira, David Fernandes, and Leandro Silva Galvão de Carvalho. 2019. Adaptação de um método preditivo para inferir o desempenho de alunos de programação. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 30. 1651.
Samuel C Fonseca, Filipe Dwan Pereira, Elaine HT Oliveira, David BF Oliveira, Leandro SG Carvalho, and Alexandra I Cristea. 2020. Automatic Subject-based Contextualisation of Programming Assignment Lists. EDM.
A. Géron. 2019. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media.
Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V. Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao. 2018. Predicting Academic Performance: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018 Companion). Association for Computing Machinery, New York, NY, USA, 175–199.
A. Kumar Veerasamy, D. D’Souza, M. V. Apiola, M. J. Laakso, and T. Salakoski. 2020. Using early assessment performance as early warning signs to identify at-risk students in programming courses. In 2020 IEEE Frontiers in Education Conference (FIE). 1–9.
Marcos Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, and Filipe Dwan Pereira. 2020. Classificação de dificuldade de questões de programação com base em métricas de código. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1323–1332.
Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 (2020), 56–67.
Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in neural information processing systems. 4765–4774.
Andersen Per-Arne, Christian Kråkevik, M. Goodwin, and A. Yazidi. 2016. Adaptive task assignment in online learning environments. ACM. Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics (2016), 5.
Filipe Pereira, Elaine Oliveira, David Fernandes, Leandro Silva G. Carvalho de Carvalho, and Hermino Junior. 2019. Otimização e automação da predição precoce do desempenho de alunos que utilizam juízes online: uma abordagem com algoritmo genético. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 30. 1451.
Filipe Dwan Pereira, Linnik Maciel de Souza, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Predição de desempenho em ambientes computacionais para turmas de programação: um Mapeamento Sistemático da Literatura. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1673–1682.
Filipe Dwan Pereira, Samuel C Fonseca, Elaine HT Oliveira, David BF Oliveira, Alexandra I Cristea, and Leandro SG Carvalho. 2020. Deep learning for early performance prediction of introductory programming students: a comparative and explanatory study. Brazilian journal of computers in education. 28 (2020), 723–749.
Filipe D Pereira, Hermino Junior, Luiz Rodriguez, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, David Oliveira, Leandro Carvalho, Samuel Fonseca, Ahmed Alamri, and Seiji Isotani. 2021. A recommender system based on effort: towards minimising negative affects and maximising achievement in CS1 learning. In International Conference on Intelligent Tutoring Systems. Springer.
Filipe D Pereira, Elaine Oliveira, Alexandra Cristea, David Fernandes, Luciano Silva, Gene Aguiar, Ahmed Alamri, and Mohammad Alshehri. 2019. Early dropout prediction for programming courses supported by online judges. In International Conference on Artificial Intelligence in Education. Springer, 67–72.
F. D. Pereira, E. H. Oliveira, D. Fernandes, and A. Cristea. 2019. Early performance prediction for CS1 course students using a combination of machine learning and an evolutionary algorithm. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Vol. 2161. IEEE, 183–184.
Filipe D Pereira, Elaine HT Oliveira, David BF Oliveira, Alexandra I Cristea, Leandro SG Carvalho, Samuel C Fonseca, Armando Toda, and Seiji Isotani. 2020. Using learning analytics in the Amazonas: understanding students’ behaviour in introductory programming. British Journal of Educational Technology (2020).
Filipe Dwan Pereira, Elaine H T Oliveira, and David F B Oliveira. 2018. Uso de um método preditivo para inferir a zona de aprendizagem de alunos de programação em um ambiente de correção automática de código. Mestrado em Informática. Universidade Federal do Amazonas, Manaus.
Filipe D Pereira, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, Seiji Isotani, Dion Laranjeira, Adriano Almeida, and Jonas Mendonça. 2020. Can We Use Gamification to Predict Students’ Performance? A Case Study Supported by an Online Judge. In International Conference on Intelligent Tutoring Systems. Springer, 259–269.
Keith Quille and Susan Bergin. 2019. CS1: how will they do? How can we help? A decade of research and practice. Computer Science Education 29, 2-3 (2019), 254–282.
Anthony Robins. 2010. Learning edge momentum: A new account of outcomes in CS1. Computer Science Education 20, 1 (2010), 37–71.
A. V. Robins. 2019. Novice programmers and introductory programming. In The Cambridge Handbook of Computing Education Research. Cambridge University Press, Cambridge, Chapter 12, 327–376.
Cristóbal Romero and Sebastian Ventura. 2019. Guest editorial: special issue on early prediction and supporting of learning performance. IEEE Transactions on Learning Technologies 12, 2 (2019), 145–147.
Christopher Watson and Frederick WB Li. 2014. Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation & technology in computer science education. 39–44.
Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Marcela Pessoa, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2020. Uma análise orientada a dados para avaliar o impacto da gamificação de um juiz on-line no desempenho de estudantes. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 491–500.
Ada Araujo, Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2021. Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 123–131.
Jens Bennedsen and Michael E Caspersen. 2019. Failure rates in introductory programming: 12 years later. ACM inroads 10, 2 (2019), 30–36.
Ryan Bockmon, Stephen Cooper, Jonathan Gratch, Jian Zhang, and Mohsen Dorodchi. 2020. Can Students’ Spatial Skills Predict Their Programming Abilities? (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 446–451.
Adam Carter, Christopher Hundhausen, and Daniel Olivares. 2019. Leveraging the IDE for learning analytics. The Cambridge Handbook of Computing Education Research, Sally A. Fincher and Anthony V. Robins (Eds.). Cambridge University Press, Cambridge (2019), 679–706.
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785 794.
Evandro B Costa, Baldoino Fonseca, Marcelo Almeida Santana, Fabrísia Ferreira de Araújo, and Joilson Rego. 2017. Evaluating the effectiveness of educational data mining techniques for early prediction of students’ academic failure in introductory programming courses. Computers in Human Behavior 73 (2017), 247–256.
Hermino Barbosa de Freitas Júnior, Filipe Dwan Pereira, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Recomendação Automática de Problemas em Juízes Online Usando Processamento de Linguagem Natural e Análise Dirigida aos Dados. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1152–1161.
Marcos Avner Pimenta de Lima Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Filipe Dwan Pereira. 2021. Uso de atributos de código para classificação da facilidade de questões de codificação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 113–122.
Joseph de Oliveira, Felipe Salem, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, and Filipe Dwan Pereira. 2020. Os estudantes leem as mensagens de feedback estendido exibidas em juízes online?. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1723–1732.
Ingrid Lima dos Santos, David Braga Fernandes Oliveira, Leandro Silva Galvão de Carvalho, Filipe Dwan Pereira, and Elaine Harada Teixeira de Oliveira. 2020. Tempos de Transição em Estados de Corretude e Erro como Indicadores de Desempenho em Juízes Online. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1283–1292.
Filipe Dwan, Elaine Oliveira, and David Fernandes. 2017. Predição de zona de aprendizagem de alunos de introdução à programação em ambientes de correção automática de código. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 28. 1507.
Samuel Fonseca, Elaine Oliveira, Filipe Pereira, David Fernandes, and Leandro Silva Galvão de Carvalho. 2019. Adaptação de um método preditivo para inferir o desempenho de alunos de programação. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 30. 1651.
Samuel C Fonseca, Filipe Dwan Pereira, Elaine HT Oliveira, David BF Oliveira, Leandro SG Carvalho, and Alexandra I Cristea. 2020. Automatic Subject-based Contextualisation of Programming Assignment Lists. EDM.
A. Géron. 2019. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media.
Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V. Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao. 2018. Predicting Academic Performance: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (Larnaca, Cyprus) (ITiCSE 2018 Companion). Association for Computing Machinery, New York, NY, USA, 175–199.
A. Kumar Veerasamy, D. D’Souza, M. V. Apiola, M. J. Laakso, and T. Salakoski. 2020. Using early assessment performance as early warning signs to identify at-risk students in programming courses. In 2020 IEEE Frontiers in Education Conference (FIE). 1–9.
Marcos Lima, Leandro Silva Galvão de Carvalho, Elaine Harada Teixeira de Oliveira, David Braga Fernandes Oliveira, and Filipe Dwan Pereira. 2020. Classificação de dificuldade de questões de programação com base em métricas de código. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1323–1332.
Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 (2020), 56–67.
Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in neural information processing systems. 4765–4774.
Andersen Per-Arne, Christian Kråkevik, M. Goodwin, and A. Yazidi. 2016. Adaptive task assignment in online learning environments. ACM. Proceedings of the 6th International Conference on Web Intelligence, Mining and Semantics (2016), 5.
Filipe Pereira, Elaine Oliveira, David Fernandes, Leandro Silva G. Carvalho de Carvalho, and Hermino Junior. 2019. Otimização e automação da predição precoce do desempenho de alunos que utilizam juízes online: uma abordagem com algoritmo genético. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 30. 1451.
Filipe Dwan Pereira, Linnik Maciel de Souza, Elaine Harada Teixeira de Oliveira, David Braga Fernandes de Oliveira, and Leandro Silva Galvão de Carvalho. 2020. Predição de desempenho em ambientes computacionais para turmas de programação: um Mapeamento Sistemático da Literatura. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 1673–1682.
Filipe Dwan Pereira, Samuel C Fonseca, Elaine HT Oliveira, David BF Oliveira, Alexandra I Cristea, and Leandro SG Carvalho. 2020. Deep learning for early performance prediction of introductory programming students: a comparative and explanatory study. Brazilian journal of computers in education. 28 (2020), 723–749.
Filipe D Pereira, Hermino Junior, Luiz Rodriguez, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, David Oliveira, Leandro Carvalho, Samuel Fonseca, Ahmed Alamri, and Seiji Isotani. 2021. A recommender system based on effort: towards minimising negative affects and maximising achievement in CS1 learning. In International Conference on Intelligent Tutoring Systems. Springer.
Filipe D Pereira, Elaine Oliveira, Alexandra Cristea, David Fernandes, Luciano Silva, Gene Aguiar, Ahmed Alamri, and Mohammad Alshehri. 2019. Early dropout prediction for programming courses supported by online judges. In International Conference on Artificial Intelligence in Education. Springer, 67–72.
F. D. Pereira, E. H. Oliveira, D. Fernandes, and A. Cristea. 2019. Early performance prediction for CS1 course students using a combination of machine learning and an evolutionary algorithm. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Vol. 2161. IEEE, 183–184.
Filipe D Pereira, Elaine HT Oliveira, David BF Oliveira, Alexandra I Cristea, Leandro SG Carvalho, Samuel C Fonseca, Armando Toda, and Seiji Isotani. 2020. Using learning analytics in the Amazonas: understanding students’ behaviour in introductory programming. British Journal of Educational Technology (2020).
Filipe Dwan Pereira, Elaine H T Oliveira, and David F B Oliveira. 2018. Uso de um método preditivo para inferir a zona de aprendizagem de alunos de programação em um ambiente de correção automática de código. Mestrado em Informática. Universidade Federal do Amazonas, Manaus.
Filipe D Pereira, Armando Toda, Elaine HT Oliveira, Alexandra I Cristea, Seiji Isotani, Dion Laranjeira, Adriano Almeida, and Jonas Mendonça. 2020. Can We Use Gamification to Predict Students’ Performance? A Case Study Supported by an Online Judge. In International Conference on Intelligent Tutoring Systems. Springer, 259–269.
Keith Quille and Susan Bergin. 2019. CS1: how will they do? How can we help? A decade of research and practice. Computer Science Education 29, 2-3 (2019), 254–282.
Anthony Robins. 2010. Learning edge momentum: A new account of outcomes in CS1. Computer Science Education 20, 1 (2010), 37–71.
A. V. Robins. 2019. Novice programmers and introductory programming. In The Cambridge Handbook of Computing Education Research. Cambridge University Press, Cambridge, Chapter 12, 327–376.
Cristóbal Romero and Sebastian Ventura. 2019. Guest editorial: special issue on early prediction and supporting of learning performance. IEEE Transactions on Learning Technologies 12, 2 (2019), 145–147.
Christopher Watson and Frederick WB Li. 2014. Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation & technology in computer science education. 39–44.
Daniel Lopes Zordan Filho, Elaine Harada Teixeira de Oliveira, Leandro Silva Galvão de Carvalho, Marcela Pessoa, Filipe Dwan Pereira, and David Braga Fernandes de Oliveira. 2020. Uma análise orientada a dados para avaliar o impacto da gamificação de um juiz on-line no desempenho de estudantes. In Anais do XXXI Simpósio Brasileiro de Informática na Educação. SBC, 491–500.
Publicado
26/04/2021
Como Citar
PEREIRA, Filipe Dwan; OLIVEIRA, Elaine Harada Teixeira de; OLIVEIRA, David Braga Fernandes de; CARVALHO, Leandro Silva Galvão de; CRISTEA, Alexandra Ioana.
Interpretable AI to Understand Early Effective and Ineffective Programming Behaviours from CS1 Learners. In: LABORATÓRIO DE IDEIAS - SIMPÓSIO BRASILEIRO DE EDUCAÇÃO EM COMPUTAÇÃO (EDUCOMP), 1. , 2021, On-line.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 16-17.
DOI: https://doi.org/10.5753/educomp_estendido.2021.14853.
