Exploring the use of open-source LLMs for Game Learning Analytics: an empirical study
Resumo
Game Learning Analytics (GLA) is essential for analyzing player behavior and identifying their learning progress through data collection and analysis. However, data modeling for GLA is a complex and challenging process. Large Language Models (LLMs) can be an alternative, especially opensource models with no usage limitations. This study investigates how the models “Gemma 2 (9B),” “Qwen 2.5 (14B),” “LLaMA 3.1 Instruct (8B),” and “Phi4 Mini (3.8B)” are capable of modeling data and filling out the data capture template from the GLBoard model. Seven GLA experts evaluated each model’s responses and identified the “Qwen” model as the most satisfactory, highlighting the potential of open-source LLMs in GLA activities.Referências
Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Fernández-Manjón, B. (2022). Game learning analytics:: Blending visual and data mining techniques to improve serious games and to better understand player learning. Journal of Learning Analytics, 9(3):32–49.
Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Manjón, B. F. (2021). Data science meets standardized game learning analytics. In 2021 IEEE Global Engineering Education Conference (EDUCON), pages 1546–1552. IEEE.
Alonso-Fernández, C., Cano, A. R., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Fernández-Manjón, B. (2019). Lessons learned applying learning analytics to assess serious games. Computers in Human Behavior, 99:301–309.
Banihashem, S. K., Dehghanzadeh, H., Clark, D., Noroozi, O., and Biemans, H. J. (2024). Learning analytics for online game-based learning: A systematic literature review. Behaviour & Information Technology, 43(12):2689–2716.
Bardin, L. (2015). Análise de conteúdo (la reto & a. pinheiro, tradução)(6ª edição). Lisboa, Portugal: Edições, 70.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Din, S. U., Baig, M. Z., and Khan, M. K. (2023). Serious games: An updated systematic literature review. arXiv preprint arXiv:2306.03098.
Freire, M., Serrano-Laguna, Á., Manero, B., Martínez-Ortiz, I., Moreno-Ger, P., and Fernández-Manjón, B. (2016). Game learning analytics: Learning analytics for serious games. In Learning, design, and technology, pages 1–29. Springer Nature Switzerland AG.
Hauge, J. B., Berta, R., Fiucci, G., Manjón, B. F., Padrón-Nápoles, C., Westra, W., and Nadolski, R. (2014). Implications of learning analytics for serious game design. In 2014 IEEE 14th international conference on advanced learning technologies, pages 230–232. IEEE.
Honda, F., Pessoa, M., Pires, F., and Harada, E. (2025a). Chatgpt, gemini or deepseek? an empirical study in game learning analytics. In Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames). SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. (2024). Building a specialist agent to assist in the implementation of game learning analytics techniques. In Simpósio Brasileiro de Informática na Educação (SBIE), pages 2856–2865. SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. T. (2025b). Challenges in educational game data modeling from the perspective of computing students: an empirical study. In Workshop sobre Educação em Computação (WEI). SBC.
IBM (2023). Ai hallucinations. [link]. Accessed: 2025-06-11.
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023). Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274.
Kitto, K., Whitmer, J., Silvers, A., and Webb, M. (2020). Creating data for learning analytics ecosystems.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
Lo, L. S. (2023). The art and science of prompt engineering: a new literacy in the information age. Internet Reference Services Quarterly, 27(4):203–210.
Macena, J., Honda, F., Melo, D., Pires, F., Oliveira, E. H., Fernandes, D., and Pessoa, M. (2024). Desafios na implementação de técnicas de gla em um jogo educacional de algoritmos: um estudo de caso. In Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames), pages 814–825. SBC.
Melo, D., Melo, R., Bernardo, J. R. S., Pessoa, M., Rodríguez, L. C., and Pires, F. (2020). Uma estratégia de game learning analytics para avaliar level design em um jogo educacional. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 622–631. SBC.
Misiejuk, K., López-Pernas, S., Kaliisa, R., and Saqr, M. (2025). Mapping the landscape of generative artificial intelligence in learning analytics: A systematic literature review. Journal of Learning Analytics, pages 1–20.
Mitsea, E., Drigas, A., and Skianis, C. (2025). A systematic review of serious games in the era of artificial intelligence, immersive technologies, the metaverse, and neurotechnologies: Transformation through meta-skills training. Electronics, 14(4):649.
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.
Shin, J., Tang, C., Mohati, T., Nayebi, M., Wang, S., and Hemmati, H. (2023). Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks. arXiv preprint arXiv:2310.10508.
Silva, D., Melo, R., Pires, F., and Pessoa, M. (2021). Avaliacão de objetos digitais de aprendizagem: como os licenciados em computação analisam jogos educacionais? Revista Novas Tecnologias na Educação, 19(2):111–121.
Silva, D., Pires, F., Melo, R., and Pessoa, M. (2022). Glboard: um sistema para auxiliar na captura e análise de dados em jogos educacionais. In Anais Estendidos do XXI Simpósio Brasileiro de Jogos e Entretenimento Digital, pages 959–968. SBC.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022). Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
Van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless. EDUCAUSE review, 41(2):16.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
Ye, Q., Axmed, M., Pryzant, R., and Khani, F. (2023). Prompt engineering a prompt engineer. arXiv preprint arXiv:2311.05661.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. (2023). Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Manjón, B. F. (2021). Data science meets standardized game learning analytics. In 2021 IEEE Global Engineering Education Conference (EDUCON), pages 1546–1552. IEEE.
Alonso-Fernández, C., Cano, A. R., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., and Fernández-Manjón, B. (2019). Lessons learned applying learning analytics to assess serious games. Computers in Human Behavior, 99:301–309.
Banihashem, S. K., Dehghanzadeh, H., Clark, D., Noroozi, O., and Biemans, H. J. (2024). Learning analytics for online game-based learning: A systematic literature review. Behaviour & Information Technology, 43(12):2689–2716.
Bardin, L. (2015). Análise de conteúdo (la reto & a. pinheiro, tradução)(6ª edição). Lisboa, Portugal: Edições, 70.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Din, S. U., Baig, M. Z., and Khan, M. K. (2023). Serious games: An updated systematic literature review. arXiv preprint arXiv:2306.03098.
Freire, M., Serrano-Laguna, Á., Manero, B., Martínez-Ortiz, I., Moreno-Ger, P., and Fernández-Manjón, B. (2016). Game learning analytics: Learning analytics for serious games. In Learning, design, and technology, pages 1–29. Springer Nature Switzerland AG.
Hauge, J. B., Berta, R., Fiucci, G., Manjón, B. F., Padrón-Nápoles, C., Westra, W., and Nadolski, R. (2014). Implications of learning analytics for serious game design. In 2014 IEEE 14th international conference on advanced learning technologies, pages 230–232. IEEE.
Honda, F., Pessoa, M., Pires, F., and Harada, E. (2025a). Chatgpt, gemini or deepseek? an empirical study in game learning analytics. In Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames). SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. (2024). Building a specialist agent to assist in the implementation of game learning analytics techniques. In Simpósio Brasileiro de Informática na Educação (SBIE), pages 2856–2865. SBC.
Honda, F., Pires, F., Pessoa, M., and Oliveira, E. H. T. (2025b). Challenges in educational game data modeling from the perspective of computing students: an empirical study. In Workshop sobre Educação em Computação (WEI). SBC.
IBM (2023). Ai hallucinations. [link]. Accessed: 2025-06-11.
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023). Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274.
Kitto, K., Whitmer, J., Silvers, A., and Webb, M. (2020). Creating data for learning analytics ecosystems.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
Lo, L. S. (2023). The art and science of prompt engineering: a new literacy in the information age. Internet Reference Services Quarterly, 27(4):203–210.
Macena, J., Honda, F., Melo, D., Pires, F., Oliveira, E. H., Fernandes, D., and Pessoa, M. (2024). Desafios na implementação de técnicas de gla em um jogo educacional de algoritmos: um estudo de caso. In Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames), pages 814–825. SBC.
Melo, D., Melo, R., Bernardo, J. R. S., Pessoa, M., Rodríguez, L. C., and Pires, F. (2020). Uma estratégia de game learning analytics para avaliar level design em um jogo educacional. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 622–631. SBC.
Misiejuk, K., López-Pernas, S., Kaliisa, R., and Saqr, M. (2025). Mapping the landscape of generative artificial intelligence in learning analytics: A systematic literature review. Journal of Learning Analytics, pages 1–20.
Mitsea, E., Drigas, A., and Skianis, C. (2025). A systematic review of serious games in the era of artificial intelligence, immersive technologies, the metaverse, and neurotechnologies: Transformation through meta-skills training. Electronics, 14(4):649.
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.
Shin, J., Tang, C., Mohati, T., Nayebi, M., Wang, S., and Hemmati, H. (2023). Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks. arXiv preprint arXiv:2310.10508.
Silva, D., Melo, R., Pires, F., and Pessoa, M. (2021). Avaliacão de objetos digitais de aprendizagem: como os licenciados em computação analisam jogos educacionais? Revista Novas Tecnologias na Educação, 19(2):111–121.
Silva, D., Pires, F., Melo, R., and Pessoa, M. (2022). Glboard: um sistema para auxiliar na captura e análise de dados em jogos educacionais. In Anais Estendidos do XXI Simpósio Brasileiro de Jogos e Entretenimento Digital, pages 959–968. SBC.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022). Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
Van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless. EDUCAUSE review, 41(2):16.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
Ye, Q., Axmed, M., Pryzant, R., and Khani, F. (2023). Prompt engineering a prompt engineer. arXiv preprint arXiv:2311.05661.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. (2023). Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
Publicado
24/11/2025
Como Citar
FILHO, Defala; HONDA, Fabrizio; PIRES, Fernanda; PESSOA, Marcela.
Exploring the use of open-source LLMs for Game Learning Analytics: an empirical study. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 36. , 2025, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 1302-1316.
DOI: https://doi.org/10.5753/sbie.2025.12885.
