ChatGPT, Gemini or DeepSeek? An empirical study in Game Learning Analytics

• Fabrizio Honda UEA / UFAM
• Marcela Pessoa UEA
• Fernanda Pires UEA
• Elaine H. T. Oliveira UFAM

DOI: https://doi.org/10.5753/sbgames.2025.10316

Resumo

Introduction: Generative artificial intelligence (GenAI), through large language models (LLMs), is an emerging area that has transformed the way of solving problems in several areas of knowledge. This technology can also offer solutions to challenges still open in Game Learning Analytics (GLA). Modeling data for GLA and implementing data capture and analysis techniques in educational games to identify evidence of learning are not trivial tasks. The use of LLMs can bring benefits; however, the wide variety of models available, which work in different ways, makes choosing the most appropriate model challenging. Objective: In this context, this article presents a comparative analysis of how LLMs (ChatGPT, Gemini, and DeepSeek) perform activities related to GLA, including the generation of templates for data capture in the GLBoard model under zero, one, and few-shot learning conditions. Methodology: For this, an empirical study was conducted with steps that involved the selection of models, definition of questions, construction of prompts, and collection and analysis of data. Results: The results indicated that ChatGPT and DeepSeek presented more accurate responses under the fewshot learning condition, with DeepSeek standing out in both GLA activities.

Palavras-chave: Generative Artificial Intelligence, Large Language Models, Game Learning Analytics, Educational Games, Prompt Engineering

Referências

Akter, S. N., Yu, Z., Muhamed, A., Ou, T., Bäuerle, A., Cabrera, Á. A., Dholakia, K., Xiong, C., e Neubig, G. (2023). An in-depth look at gemini’s language abilities. arXiv preprint arXiv:2312.11444.

AlBadarin, Y., Tukiainen, M., Saqr, M., e Pope, N. (2023). A systematic literature review of empirical research on chatgpt in education. Available at SSRN 4562771.

Alonso-Fernandez, C., Calvo, A., Freire, M., Martinez-Ortiz, I., e Fernandez-Manjon, B. (2017). Systematizing game learning analytics for serious games. In 2017 IEEE global engineering education conference (EDUCON), pages 1111–1118. IEEE.

Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., e Fernández-Manjón, B. (2022). Game learning analytics:: Blending visual and data mining techniques to improve serious games and to better understand player learning. Journal of Learning Analytics, 9(3):32–49.

Banihashem, S. K., Dehghanzadeh, H., Clark, D., Noroozi, O., e Biemans, H. J. (2024). Learning analytics for online game-based learning: A systematic literature review. Behaviour & Information Technology, 43(12):2689–2716.

BBC News (2025). DeepSeek: The Chinese AI app that has the world talking. BBC News. Accessed: 2025-02-05.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Chan, A., Kharkar, A., Moghaddam, R. Z., Mohylevskyy, Y., Helyar, A., Kamal, E., Elkamhawy, M., e Sundaresan, N. (2023). Transformer-based vulnerability detection in code at edittime: Zero-shot, few-shot, or fine-tuning? arXiv preprint arXiv:2306.01754.

DemandSage (2025). Chatgpt statistics 2025: Users, traffic, revenue more. Accessed: 2025-02-17.

Freire, M., Serrano-Laguna, Á., Manero, B., Martínez-Ortiz, I., Moreno-Ger, P., e Fernández-Manjón, B. (2016). Game learning analytics: Learning analytics for serious games. In Learning, design, and technology, pages 1–29. Springer Nature Switzerland AG.

Honda, F., Pires, F., Pessoa, M., e Oliveira, E. H. (2024). Building a specialist agent to assist in the implementation of game learning analytics techniques. In Simpósio Brasileiro de Informática na Educação (SBIE), pages 2856–2865. SBC.

IBM (2023). Ai hallucinations. [link]. Accessed: 2025-04-20.

Imran, M. e Almusharraf, N. (2024). Google gemini as a next generation ai educational tool: a review of emerging educational technology. Smart Learning Environments, 11(1):22.

Jovanovic, M. e Campbell, M. (2022). Generative artificial intelligence: Trends and prospects. Computer, 55(10):107–112.

Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023). Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274.

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., e Neubig, G. (2023a). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.

Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023b). Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology, page 100017.

Lo, L. S. (2023). The art and science of prompt engineering: a new literacy in the information age. Internet Reference Services Quarterly, 27(4):203–210.

Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

Nexocode (2023). Customizing large language models: A comprehensive guide. Accessed: 2025-02-17.

Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., e Chadha, A. (2024). A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.

Saveski, G. L., Westera, W., Yuan, L., Hollins, P., Manjón, B. F., Ger, P. M., e Stefanov, K. (2016). What serious game studios want from ict research: identifying developers’ needs. In Games and Learning Alliance: 4th International Conference, GALA 2015, Rome, Italy, December 9-11, 2015, Revised Selected Papers 4, pages 32–41. Springer.

Sengar, S. S., Hasan, A. B., Kumar, S., e Carroll, F. (2024). Generative artificial intelligence: a systematic review and applications. Multimedia Tools and Applications, pages 1–40.

SEO.ai (2025). Deepseek users, downloads, and growth. Accessed: 2025-02-17.

Silva, D., Pires, F., Melo, R., e Pessoa, M. (2022). Glboard: um sistema para auxiliar na captura e análise de dados em jogos educacionais. In Anais Estendidos do XXI Simpósio Brasileiro de Jogos e Entretenimento Digital, pages 959–968. SBC.

Singh, H. e Singh, A. (2023). Chatgpt: Systematic review, applications, and agenda for multidisciplinary research. Journal of Chinese economic and business studies, 21(2):193–212.

Ye, Q., Axmed, M., Pryzant, R., e Khani, F. (2023). Prompt engineering a prompt engineer. arXiv preprint arXiv:2311.05661.

Zamfirescu-Pereira, J., Wong, R. Y., Hartmann, B., e Yang, Q. (2023). Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–21.