Towards Interpretability of Attention-Based Knowledge Tracing Models
Resumo
Knowledge Tracing (KT) models based on attention mechanisms have demonstrated in literature the capability to predict student performance more accurately than previous models in some datasets. However, they fail to directly infer student knowledge. In this paper, we apply a proposed extension already seen in KT literature in order to infer latent knowledge to these models. We apply the extension to four different attention-based KT models, to investigate whether these models can better infer the knowledge outside the learning system than previous models. We find that attention-based models can generate better knowledge estimate correlations with student’s scores than the previous models.
Palavras-chave:
Deep knowledge tracing, Attention-based knowledge tracing, Adaptive learning
Referências
Abadi, Martin & Agarwal, Ashish & Barham, Paul & Brevdo, Eugene & Chen, Zhifeng & Citro, Craig & Corrado, G.s & Davis, Andy & Dean, Jeffrey & Devin, Matthieu & Ghemawat, Sanjay & Goodfellow, Ian & Harp, Andrew & Irving, Geoffrey & Isard, Michael & Jia, Yangqing & Kaiser, Lukasz & Kudlur, Manjunath & Levenberg, Josh & Zheng, Xiaoqiang. (2015). TensorFlow : Large-Scale Machine Learning on Heterogeneous Distributed Systems.
Akyuz, Y. (2020) Effects of Intelligent Tutoring Systems (ITS) on Personalized Learning (PL). Creative Education, 11, 953-978.
Cao J, Yang T, Lai IK-W, Wu J. Student acceptance of intelligent tutoring systems during COVID-19: The effect of political influence. International Social Work. 2021;0(0):129-136. doi:10.1177/0020872809348959
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C. & Zhang, Z. (2015). MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems.. CoRR, abs/1512.01274.
Corbett, A.T., Anderson, J.R. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model User-Adap Inter 4, 253–278 (1994).
Corbett, A.T., & Bhatnagar, A. (1997). Student Modeling in the ACT Programming Tutor: Adjusting a Procedural Learning Model With Declarative Knowledge.
Ding, Xinyi & Larson, Eric. (2021). On the interpretability of deep learning based models for knowledge tracing.
Gervet, T., Koedinger, K., Schneider, J., and Mitchell, T. (2020). When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining, 12(3):31–54.
Ghosh, A., Heffernan, N. T., and Lan, A. S. (2020). Context-aware attentive knowledge tracing. CoRR, abs/2007.12324.
Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. In arXiv:1410.5401.
Guo, X., Huang, Z., Gao, J., Shang, M., Shu, M., and Sun, J. (2021). Enhancing knowledge tracing via adversarial training. In Proceedings of the 29th ACM International Conference on Multimedia, pages 367–375.
Kenton, J. D. M.-W. C. and Toutanova, L. K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
Liu, Q., Shen, S., Huang, Z., Chen, E., and Zheng, Y. (2021). A survey of knowledge tracing. In arXiv:2105.15106.
Lu, Y., Wang, D., Meng, Q., and Chen, P. (2020). Towards interpretable deep learning models for knowledge tracing. In International Conference on Artificial Intelligence in Education, pages 185–190. Springer.
Mandalapu, V., Gong, J., and Chen, L. (2021). Do we need to go deep? knowledge tracing with big data. In 35th AAAI Conference on Artificial Intelligence.
Mao, Y. (2018). Deep learning vs. bayesian knowledge tracing: Student models for interventions. Journal of educational data mining, 10(2).
Pandey, S. and Karypis, G. (2019). A self-attentive model for knowledge tracing. In Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019).
Pandey, S., Karypis, G., and Srivastava, J. (2021). An empirical comparison of deep learning models for knowledge tracing on large-scale dataset. In 35th AAAI Conference on Artificial Intelligence.
Pang, T., Yang, X., Dong, Y., Su, H., and Zhu, J. (2021). Bag of tricks for adversarial training. In International Conference on Learning Representations.
Pantelimon, F.-V., Bologa, R., Toma, A., and Posedaru, B.-S. (2021). The evolution of ai-driven educational systems during the covid-19 pandemic. Sustainability, 13(23):13501.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.
Pavlik Jr, P., Cen, H., and Koedinger, K. (2009). Performance factors analysis - a new alternative to knowledge tracing. Frontiers in Artificial Intelligence and Applications, 200:531–538.
Penteado, B. and Fornazin, M. (2021). Detecção de inovações tecnológicas na evolução da informática educacional no brasil. In Anais do XXXII Simpósio Brasileiro de Informática na Educação, pages 157–167, Porto Alegre, RS, Brasil. SBC.
Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L. J., and SohlDickstein, J. (2015). Deep knowledge tracing. Advances in neural information processing systems, 28.
Raposo, A., Maranhão, D., and Neto, C. S. (2020). Análise da capacidade preditiva de técnicas para modelagem do conhecimento aplicadas ao aprendizado de algoritmos. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 1653–1662, Porto Alegre, RS, Brasil. SBC.
Richey, J. E., Andres-Bray, J. M. L., Mogessie, M., Scruggs, R., Andres, J. M., Star, J. R., Baker, R. S., and McLaren, B. M. (2019). More confusion and frustration, better learning: The impact of erroneous examples. Computers & Education, 139:173–190.
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. (2016). Metalearning with memory-augmented neural networks. In Balcan, M. F. and Weinberger, K. Q., editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1842–1850, New York, New York, USA. PMLR.
Santos, C., Paillard, G., Moreira, L., Filho, F. R. S., and Coutinho, E. (2020). Uma análise qualitativa sobre atividades remotas em disciplinas no período de isolamento social. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 292–301, Porto Alegre, RS, Brasil. SBC.
Scruggs, R., Baker, R. S., and McLaren, B. M. (2020). Extending deep knowledge tracing: Inferring interpretable knowledge and predicting post-system performance. In Proceedings of the 28th International Conference on Computers in Education.
Sukhbaatar, S., Weston, J., Fergus, R., et al. (2015). End-to-end memory networks. Advances in neural information processing systems, 28.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Wang, F., Liu, Q., Chen, E., Huang, Z., Chen, Y., Yin, Y., Huang, Z., and Wang, S. (2020). Neural cognitive diagnosis for intelligent education systems. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6153–6161.
Yeung, C.-K. and Yeung, D.-Y. (2018). Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale, pages 1–10.
Zeng, J., Zhang, Q., Xie, N., and Yang, B. (2021). Application of deep self-attention in knowledge tracing. In Arxiv:2105.07909.
Zhang, J., Shi, X., King, I., and Yeung, D.-Y. (2017). Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th international conference on World Wide Web, pages 765–774.
Zhu, J., Yu, W., Zheng, Z., Huang, C., Tang, Y., and Fung, G. P. C. (2020). Learning from interpretable analysis: Attention-based knowledge tracing. Artificial Intelligence in Education, pages 364–368.
Akyuz, Y. (2020) Effects of Intelligent Tutoring Systems (ITS) on Personalized Learning (PL). Creative Education, 11, 953-978.
Cao J, Yang T, Lai IK-W, Wu J. Student acceptance of intelligent tutoring systems during COVID-19: The effect of political influence. International Social Work. 2021;0(0):129-136. doi:10.1177/0020872809348959
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C. & Zhang, Z. (2015). MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems.. CoRR, abs/1512.01274.
Corbett, A.T., Anderson, J.R. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model User-Adap Inter 4, 253–278 (1994).
Corbett, A.T., & Bhatnagar, A. (1997). Student Modeling in the ACT Programming Tutor: Adjusting a Procedural Learning Model With Declarative Knowledge.
Ding, Xinyi & Larson, Eric. (2021). On the interpretability of deep learning based models for knowledge tracing.
Gervet, T., Koedinger, K., Schneider, J., and Mitchell, T. (2020). When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining, 12(3):31–54.
Ghosh, A., Heffernan, N. T., and Lan, A. S. (2020). Context-aware attentive knowledge tracing. CoRR, abs/2007.12324.
Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. In arXiv:1410.5401.
Guo, X., Huang, Z., Gao, J., Shang, M., Shu, M., and Sun, J. (2021). Enhancing knowledge tracing via adversarial training. In Proceedings of the 29th ACM International Conference on Multimedia, pages 367–375.
Kenton, J. D. M.-W. C. and Toutanova, L. K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
Liu, Q., Shen, S., Huang, Z., Chen, E., and Zheng, Y. (2021). A survey of knowledge tracing. In arXiv:2105.15106.
Lu, Y., Wang, D., Meng, Q., and Chen, P. (2020). Towards interpretable deep learning models for knowledge tracing. In International Conference on Artificial Intelligence in Education, pages 185–190. Springer.
Mandalapu, V., Gong, J., and Chen, L. (2021). Do we need to go deep? knowledge tracing with big data. In 35th AAAI Conference on Artificial Intelligence.
Mao, Y. (2018). Deep learning vs. bayesian knowledge tracing: Student models for interventions. Journal of educational data mining, 10(2).
Pandey, S. and Karypis, G. (2019). A self-attentive model for knowledge tracing. In Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019).
Pandey, S., Karypis, G., and Srivastava, J. (2021). An empirical comparison of deep learning models for knowledge tracing on large-scale dataset. In 35th AAAI Conference on Artificial Intelligence.
Pang, T., Yang, X., Dong, Y., Su, H., and Zhu, J. (2021). Bag of tricks for adversarial training. In International Conference on Learning Representations.
Pantelimon, F.-V., Bologa, R., Toma, A., and Posedaru, B.-S. (2021). The evolution of ai-driven educational systems during the covid-19 pandemic. Sustainability, 13(23):13501.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.
Pavlik Jr, P., Cen, H., and Koedinger, K. (2009). Performance factors analysis - a new alternative to knowledge tracing. Frontiers in Artificial Intelligence and Applications, 200:531–538.
Penteado, B. and Fornazin, M. (2021). Detecção de inovações tecnológicas na evolução da informática educacional no brasil. In Anais do XXXII Simpósio Brasileiro de Informática na Educação, pages 157–167, Porto Alegre, RS, Brasil. SBC.
Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L. J., and SohlDickstein, J. (2015). Deep knowledge tracing. Advances in neural information processing systems, 28.
Raposo, A., Maranhão, D., and Neto, C. S. (2020). Análise da capacidade preditiva de técnicas para modelagem do conhecimento aplicadas ao aprendizado de algoritmos. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 1653–1662, Porto Alegre, RS, Brasil. SBC.
Richey, J. E., Andres-Bray, J. M. L., Mogessie, M., Scruggs, R., Andres, J. M., Star, J. R., Baker, R. S., and McLaren, B. M. (2019). More confusion and frustration, better learning: The impact of erroneous examples. Computers & Education, 139:173–190.
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. (2016). Metalearning with memory-augmented neural networks. In Balcan, M. F. and Weinberger, K. Q., editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1842–1850, New York, New York, USA. PMLR.
Santos, C., Paillard, G., Moreira, L., Filho, F. R. S., and Coutinho, E. (2020). Uma análise qualitativa sobre atividades remotas em disciplinas no período de isolamento social. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 292–301, Porto Alegre, RS, Brasil. SBC.
Scruggs, R., Baker, R. S., and McLaren, B. M. (2020). Extending deep knowledge tracing: Inferring interpretable knowledge and predicting post-system performance. In Proceedings of the 28th International Conference on Computers in Education.
Sukhbaatar, S., Weston, J., Fergus, R., et al. (2015). End-to-end memory networks. Advances in neural information processing systems, 28.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Wang, F., Liu, Q., Chen, E., Huang, Z., Chen, Y., Yin, Y., Huang, Z., and Wang, S. (2020). Neural cognitive diagnosis for intelligent education systems. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6153–6161.
Yeung, C.-K. and Yeung, D.-Y. (2018). Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale, pages 1–10.
Zeng, J., Zhang, Q., Xie, N., and Yang, B. (2021). Application of deep self-attention in knowledge tracing. In Arxiv:2105.07909.
Zhang, J., Shi, X., King, I., and Yeung, D.-Y. (2017). Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th international conference on World Wide Web, pages 765–774.
Zhu, J., Yu, W., Zheng, Z., Huang, C., Tang, Y., and Fung, G. P. C. (2020). Learning from interpretable analysis: Attention-based knowledge tracing. Artificial Intelligence in Education, pages 364–368.
Publicado
16/11/2022
Como Citar
RODRIGUES, Thales B. S. F.; DE SOUZA, Jairo F.; BERNARDINO, Heder S.; BAKER, Ryan S..
Towards Interpretability of Attention-Based Knowledge Tracing Models. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 33. , 2022, Manaus.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 810-821.
DOI: https://doi.org/10.5753/sbie.2022.224685.
