Chabots with Learning Analytics Tools in Programming Education: A Systematic Mapping Study

• Aline S. O. Valente UNB
• Warley M. V. Junior Unifesspa
• Maristela Holanda UNB

DOI: https://doi.org/10.5753/educomp.2025.4904

Abstract

The use of chatbots has been applied in different areas, specifically in the area of education. One of the biggest challenges occurs within undergraduate courses, which is to train professionals who know how to use chatbots as tools to aid knowledge, not as the final answer to a problem. This article describes a systematic mapping study that seeks to characterize studies that address the use of chatbots with learning analytics tools to help teachers improve their understanding of student performance in teaching programming. Thirteen relevant articles were identified, from which important information was extracted, such as: chatGPT being the most used chatbot in teaching programming and reports as the most collected type of data. Based on this study, it was possible to obtain insights about chatbots that have learning analytics.

References

Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., and Chatzigeorgiou, A. (2019). Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology, 106:201–230.

Baker, P. (2023). ChatGPT f¨ur Dummies. John Wiley & Sons.

Becker, B. A. and Quille, K. (2019). 50 years of cs1 at sigcse: A review of the evolution of introductory programming education research. In Proceedings of the 50th acm technical symposium on computer science education, pages 338–344.

Carreira, G., Silva, L., Mendes, A. J., and Oliveira, H. G. (2022). Pyo, a chatbot assistant for introductory programming students. In 2022 International Symposium on Computers in Education (SIIE), pages 1–6. IEEE.

Chen, J., Lu, X., Du, Y., Rejtig, M., Bagley, R., Horn, M., and Wilensky, U. (2024). Learning agent-based modeling with llm companions: Experiences of novices and experts using chatgpt & netlogo chat. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–18.

Chen, Z., Xu, M., Hu, Z., Zhang, S., Zhang, J., Jiang, X., and Jumani, A. K. (2021). Multimedia educational system and its improvement using ai model for a higher education platform. Journal of Multiple-Valued Logic & Soft Computing, 36.

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6):683–695.

Deng, X. and Yu, Z. (2023). A meta-analysis and systematic review of the effect of chatbot technology use in sustainable education. Sustainability, 15(4):2940.

Dunder, N., Lundborg, S.,Wong, J., and Viberg, O. (2024). Kattis vs chatgpt: Assessment and evaluation of programming tasks in the age of artificial intelligence. In Proceedings of the 14th Learning Analytics and Knowledge Conference, pages 821–827.

Hobert, S. (2019). Say hello to ‘coding tutor’! design and evaluation of a chatbot-based learning system supporting students to learn to program.

Ilic, M., Kekovic, G., Mikic, V., Mangaroska, K., Kopanja, L., and Vesin, B. (2024). Predicting student performance in a programming tutoring system using ai and filtering techniques. IEEE Transactions on Learning Technologies.

Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., and Linkman, S. (2009). Systematic literature reviews in software engineering–a systematic literature review. Information and software technology, 51(1):7–15.

Krishnan, R., Nair, S., Saamuel, B. S., Justin, S., Iwendi, C., Biamba, C., and Ibeke, E. (2022). Smart analysis of learners performance using learning analytics for improving academic progression: a case study model. Sustainability, 14(6):3378.

Kuhail, M. A., Alturki, N., Alramlawi, S., and Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1):973–1018.

Liu, M. and M’hiri, F. (2024). Beyond traditional teaching: Large language models as simulated teaching assistants in computer science. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pages 743–749.

Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018). Introductory programming: a systematic literature review. In Proceedings companion of the 23rd annual ACM conference on innovation and technology in computer science education, pages 55–106.

Melo, B. and Sant’Ana, G. (2012). A prática da metodologia ativa. Com Ciências Saúde [Internet], 23(4):327–39.

Okonkwo, C. W. and Ade-Ibijola, A. (2020). Python-bot: A chatbot for teaching python programming. Engineering Letters, 29(1).

Ouyang, F., Guo, M., Zhang, N., Bai, X., and Jiao, P. (2024). Comparing the effects of instructor manual feedback and chatgpt intelligent feedback on collaborative programming in china’s higher education. IEEE Transactions on Learning Technologies.

Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and software technology, 64:1–18.

Raza, S. A., Qazi, Z., Qazi, W., and Ahmed, M. (2022). E-learning in higher education during covid-19: evidence from blackboard learning system. Journal of Applied Research in Higher Education, 14(4):1603–1622.

Robins, A. V. (2019). 12 novice programmers and introductory programming. The Cambridge handbook of computing education research, pages 327–376.

Shukla, V. K. and Verma, A. (2019). Enhancing lms experience through aiml base and retrieval base chatbot using r language. In 2019 International Conference on Automation, Computational and Technology Management (ICACTM), pages 561–567. IEEE.

Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10):1380–1400.

Straková, N. and Válek, J. (2024). Chatbots as a learning tool: Artificial intelligence in education. R&E-SOURCE, pages 245–265.

Stupina, M. and Paniotova, V. (2023). An educational chatbot in a blended learning environment. In 2023 3rd International Conference on Technology Enhanced Learning in Higher Education (TELE), pages 276–279. IEEE.

Tang, J., Li, B., and Li, Z. (2023). Application and exploration of artificial intelligence technology in student learning in higher education. In 2023 5th International Workshop on Artificial Intelligence and Education (WAIE), pages 60–64. IEEE.

Valoyes-Porras, D. M., Rodríguez-Obregón, J. S., Salamanca-Sánchez, D. S., and Feijóo-García, M. A. (2022). Retoñosapp: Work in progress on a platform to support the teaching of programming in cs through the automation and customization of learning processes guided by artificial intelligence. In CHIRA, pages 179–186.

Watson, C. and Li, F. W. (2014). Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation & technology in computer science education, pages 39–44.

Woerner, J. H., Turtova, A. P., and Lang, A. S. (2024). Transformative potentials and ethical considerations of ai tools in higher education: Case studies and reflections. In SoutheastCon 2024, pages 510–515. IEEE.

Zhang, R., Zou, D., and Cheng, G. (2024). A review of chatbot-assisted learning: pedagogical approaches, implementations, factors leading to effectiveness, theories, and future directions. Interactive Learning Environments, 32(8):4529–4557.