Evaluation of a specialist agent in Game Learning Analytics by learning designers: a case study
Resumo
Using Game Learning Analytics techniques, or Learning Analytics for Serious Games, provides valuable insights to stakeholders, allowing the validation of game design and the identification of learning evidence. However, learning designers consider a preliminary step complex: modeling the data to list GLA variables. Using Large Language Models is an alternative in this scenario, which motivated us to create the “GLA Specialist” – an intelligent GLA agent – in a previous work. This research consists of a case study to evaluate it with computer science students from a public university as learning designers. The results were positive, indicating good acceptance of the agent and data models compatible with the GLBoard model, along with a list of corrections.Referências
Alonso-Fernandez, C., Calvo, A., Freire, M., Martinez-Ortiz, I., and Fernandez-Manjon, B. (2017). Systematizing game learning analytics for serious games. In 2017 IEEE global engineering education conference (EDUCON), pages 1111–1118. IEEE.
Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Fernández-Manjón, B. (2022). Game learning analytics:: Blending visual and data mining techniques to improve serious games and to better understand player learning. Journal of Learning Analytics, 9(3):32–49.
Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Manjón, B. F. (2021). Data science meets standardized game learning analytics. In 2021 IEEE Global Engineering Education Conference (EDUCON), pages 1546–1552. IEEE.
Banihashem, S. K., Dehghanzadeh, H., Clark, D., Noroozi, O., and Biemans, H. J. (2024). Learning analytics for online game-based learning: A systematic literature review. Behaviour & Information Technology, 43(12):2689–2716.
Bardin, L. (2015). Análise de conteúdo (la reto & a. pinheiro, tradução)(6ª edição). Lisboa, Portugal: Edições, 70.
Cordeiro, E. A. and Duarte, E. M. (2020). Jogos educacionais digitais: estado da arte em trabalhos de conclusão de curso. Revista Sítio Novo, 4(1):125–133.
Freire, M., Serrano-Laguna, Á., Manero, B., Martínez-Ortiz, I., Moreno-Ger, P., and Fernández-Manjón, B. (2016). Game learning analytics: Learning analytics for serious games. In Learning, design, and technology, pages 1–29. Springer Nature Switzerland AG.
Hauge, J. B., Berta, R., Fiucci, G., Manjón, B. F., Padrón-Nápoles, C., Westra, W., and Nadolski, R. (2014). Implications of learning analytics for serious game design. In 2014 IEEE 14th international conference on advanced learning technologies, pages 230–232. IEEE.
Honda, F., Macena, J., Duarte, J. C., Pires, F., Pessoa, M., and Oliveira, E. H. (2023). Um estudo de caso para a implementação de game learning analytics (gla) no desenvolvimento de jogos educacionais. In Workshop de Aplicações Práticas de Learning Analytics em Instituições de Ensino no Brasil (WAPLA), pages 138–146. SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. (2024). Building a specialist agent to assist in the implementation of game learning analytics techniques. In Simpósio Brasileiro de Informática na Educação (SBIE), pages 2856–2865. SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. T. (2025). Challenges in educational game data modeling from the perspective of computing students: an empirical study. In Workshop sobre Educação em Computação (WEI). SBC.
IBM (2023). Ai hallucinations. [link]. Accessed: 2025-05-21.
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023). Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274.
Kitto, K., Whitmer, J., Silvers, A., and Webb, M. (2020). Creating data for learning analytics ecosystems.
Macena, J., Honda, F., Melo, D., Pires, F., Oliveira, E. H. T., Fernandes, D., and Pessoa, M. (2024). Desafios na implementação de técnicas de gla em um jogo educacional de algoritmos: um estudo de caso. In Anais Estendidos do XXIII Simpósio Brasileiro de Jogos e Entretenimento Digital. SBC.
Merikko, J. and Silvola, A. (2024). An ai agent facilitating student help-seeking: Producing data on student support needs. In LAK 2024 Workshops: co-located with 14th International Conference on Learning Analytics and Knowledge (LAK 2024), pages 185–194. CEUR-WS. org.
Misiejuk, K., López-Pernas, S., Kaliisa, R., and Saqr, M. (2025). Mapping the landscape of generative artificial intelligence in learning analytics: A systematic literature review. Journal of Learning Analytics, pages 1–20.
Mitsea, E., Drigas, A., and Skianis, C. (2025). A systematic review of serious games in the era of artificial intelligence, immersive technologies, the metaverse, and neurotechnologies: Transformation through meta-skills training. Electronics, 14(4):649.
Mostachetti, I., Vitali, A., Regazzoni, D., Rizzi, C., and Salvi, G. P. (2025). Llm-driven adjustments in serious games: A feasibility analysis. Studies in health technology and informatics, 324:164–169.
Nascimento, A. M. d. S. and Leite Bruno, S. (2024). Jogos digitais educacionais: o que as dissertações e teses dizem? International Journal Education and Teaching (PDVL), 7(2):62–87.
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435.
Portella, F. F., Tubelo, R. A., Zanatta, E. J., and Pinto, M. E. B. (2017). Experiência da una-sus/ufcspa no desenvolvimento de jogos educacionais. Experiências exitosas da Rede UNA-SUS, page 196.
Runeson, P., Host, M., Rainer, A., and Regnell, B. (2012). Case study research in software engineering: Guidelines and examples. John Wiley & Sons.
Saveski, G. L., Westera, W., Yuan, L., Hollins, P., Manjón, B. F., Ger, P. M., and Stefanov, K. (2016). What serious game studios want from ict research: identifying developers’ needs. In Games and Learning Alliance: 4th International Conference, GALA 2015, Rome, Italy, December 9-11, 2015, Revised Selected Papers 4, pages 32–41. Springer.
Silva, D., Melo, R., Pires, F., and Pessoa, M. (2021). Avaliacão de objetos digitais de aprendizagem: como os licenciados em computação analisam jogos educacionais? Revista Novas Tecnologias na Educação, 19(2):111–121.
Silva, D., Pires, F., Melo, R., and Pessoa, M. (2022). Glboard: um sistema para auxiliar na captura e análise de dados em jogos educacionais. In Anais Estendidos do XXI Simpósio Brasileiro de Jogos e Entretenimento Digital, pages 959–968. SBC.
Tarouco, L. M. R., Konrath, M. L. P., and da Silva Grando, A. R. (2005). O aluno como co-construtor e desenvolvedor de jogos educacionais. Revista Novas Tecnologias na Educação, 3(2).
Wang, T., Zhan, Y., Lian, J., Hu, Z., Yuan, N. J., Zhang, Q., Xie, X., and Xiong, H. (2025). Llm-powered multi-agent framework for goal-oriented learning in intelligent tutoring system. arXiv preprint arXiv:2501.15749.
Wohlin, C. (2021). Case study research in software engineering—it is a case, and it is a study, but is it a case study? Information and Software Technology, 133:106514.
Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Fernández-Manjón, B. (2022). Game learning analytics:: Blending visual and data mining techniques to improve serious games and to better understand player learning. Journal of Learning Analytics, 9(3):32–49.
Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Manjón, B. F. (2021). Data science meets standardized game learning analytics. In 2021 IEEE Global Engineering Education Conference (EDUCON), pages 1546–1552. IEEE.
Banihashem, S. K., Dehghanzadeh, H., Clark, D., Noroozi, O., and Biemans, H. J. (2024). Learning analytics for online game-based learning: A systematic literature review. Behaviour & Information Technology, 43(12):2689–2716.
Bardin, L. (2015). Análise de conteúdo (la reto & a. pinheiro, tradução)(6ª edição). Lisboa, Portugal: Edições, 70.
Cordeiro, E. A. and Duarte, E. M. (2020). Jogos educacionais digitais: estado da arte em trabalhos de conclusão de curso. Revista Sítio Novo, 4(1):125–133.
Freire, M., Serrano-Laguna, Á., Manero, B., Martínez-Ortiz, I., Moreno-Ger, P., and Fernández-Manjón, B. (2016). Game learning analytics: Learning analytics for serious games. In Learning, design, and technology, pages 1–29. Springer Nature Switzerland AG.
Hauge, J. B., Berta, R., Fiucci, G., Manjón, B. F., Padrón-Nápoles, C., Westra, W., and Nadolski, R. (2014). Implications of learning analytics for serious game design. In 2014 IEEE 14th international conference on advanced learning technologies, pages 230–232. IEEE.
Honda, F., Macena, J., Duarte, J. C., Pires, F., Pessoa, M., and Oliveira, E. H. (2023). Um estudo de caso para a implementação de game learning analytics (gla) no desenvolvimento de jogos educacionais. In Workshop de Aplicações Práticas de Learning Analytics em Instituições de Ensino no Brasil (WAPLA), pages 138–146. SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. (2024). Building a specialist agent to assist in the implementation of game learning analytics techniques. In Simpósio Brasileiro de Informática na Educação (SBIE), pages 2856–2865. SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. T. (2025). Challenges in educational game data modeling from the perspective of computing students: an empirical study. In Workshop sobre Educação em Computação (WEI). SBC.
IBM (2023). Ai hallucinations. [link]. Accessed: 2025-05-21.
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023). Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274.
Kitto, K., Whitmer, J., Silvers, A., and Webb, M. (2020). Creating data for learning analytics ecosystems.
Macena, J., Honda, F., Melo, D., Pires, F., Oliveira, E. H. T., Fernandes, D., and Pessoa, M. (2024). Desafios na implementação de técnicas de gla em um jogo educacional de algoritmos: um estudo de caso. In Anais Estendidos do XXIII Simpósio Brasileiro de Jogos e Entretenimento Digital. SBC.
Merikko, J. and Silvola, A. (2024). An ai agent facilitating student help-seeking: Producing data on student support needs. In LAK 2024 Workshops: co-located with 14th International Conference on Learning Analytics and Knowledge (LAK 2024), pages 185–194. CEUR-WS. org.
Misiejuk, K., López-Pernas, S., Kaliisa, R., and Saqr, M. (2025). Mapping the landscape of generative artificial intelligence in learning analytics: A systematic literature review. Journal of Learning Analytics, pages 1–20.
Mitsea, E., Drigas, A., and Skianis, C. (2025). A systematic review of serious games in the era of artificial intelligence, immersive technologies, the metaverse, and neurotechnologies: Transformation through meta-skills training. Electronics, 14(4):649.
Mostachetti, I., Vitali, A., Regazzoni, D., Rizzi, C., and Salvi, G. P. (2025). Llm-driven adjustments in serious games: A feasibility analysis. Studies in health technology and informatics, 324:164–169.
Nascimento, A. M. d. S. and Leite Bruno, S. (2024). Jogos digitais educacionais: o que as dissertações e teses dizem? International Journal Education and Teaching (PDVL), 7(2):62–87.
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435.
Portella, F. F., Tubelo, R. A., Zanatta, E. J., and Pinto, M. E. B. (2017). Experiência da una-sus/ufcspa no desenvolvimento de jogos educacionais. Experiências exitosas da Rede UNA-SUS, page 196.
Runeson, P., Host, M., Rainer, A., and Regnell, B. (2012). Case study research in software engineering: Guidelines and examples. John Wiley & Sons.
Saveski, G. L., Westera, W., Yuan, L., Hollins, P., Manjón, B. F., Ger, P. M., and Stefanov, K. (2016). What serious game studios want from ict research: identifying developers’ needs. In Games and Learning Alliance: 4th International Conference, GALA 2015, Rome, Italy, December 9-11, 2015, Revised Selected Papers 4, pages 32–41. Springer.
Silva, D., Melo, R., Pires, F., and Pessoa, M. (2021). Avaliacão de objetos digitais de aprendizagem: como os licenciados em computação analisam jogos educacionais? Revista Novas Tecnologias na Educação, 19(2):111–121.
Silva, D., Pires, F., Melo, R., and Pessoa, M. (2022). Glboard: um sistema para auxiliar na captura e análise de dados em jogos educacionais. In Anais Estendidos do XXI Simpósio Brasileiro de Jogos e Entretenimento Digital, pages 959–968. SBC.
Tarouco, L. M. R., Konrath, M. L. P., and da Silva Grando, A. R. (2005). O aluno como co-construtor e desenvolvedor de jogos educacionais. Revista Novas Tecnologias na Educação, 3(2).
Wang, T., Zhan, Y., Lian, J., Hu, Z., Yuan, N. J., Zhang, Q., Xie, X., and Xiong, H. (2025). Llm-powered multi-agent framework for goal-oriented learning in intelligent tutoring system. arXiv preprint arXiv:2501.15749.
Wohlin, C. (2021). Case study research in software engineering—it is a case, and it is a study, but is it a case study? Information and Software Technology, 133:106514.
Publicado
24/11/2025
Como Citar
HONDA, Fabrizio; PESSOA, Marcela; HARADA, Elaine; PIRES, Fernanda.
Evaluation of a specialist agent in Game Learning Analytics by learning designers: a case study. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 36. , 2025, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 1361-1375.
DOI: https://doi.org/10.5753/sbie.2025.12894.
