A Comparative Analysis of Reinforcement Learning Training Strategies for Agents on Competitive 2v2 Soccer

• Marcelo P. Rezende UFPel
• Rafael P. Torchelsen UFPel / UFRGS

DOI: https://doi.org/10.5753/sbgames.2026.24042

Resumo

Introduction: The use of Reinforcement Learning (RL) in digital games has grown significantly, enabling agents to learn complex behaviors through interaction, with many approaches emphasizing improvements in agent performance. However, in game development, aspects such as perceived difficulty, behavioral naturalness, and player enjoyment are equally important. Objective: This study aimed to analyze the impact of different RL training strategies on both objective performance and subjective player experience in a competitive game scenario. Methodology or Steps: A 2v2 soccer environment was developed in Unity using the ML-Agents toolkit. The original scene was modified to include deterministic agents with scalable difficulty. Three models were trained using the multi-agent Posthumous Credit Assignment (POCA) algorithm: one with self-play and two against deterministic opponents with difficulty adjustment based on win streak and win rate. For evaluation, pairs of human players participated in sessions of three matches, each against a different model, followed by a questionnaire assessing perceived difficulty, movement naturalness, and enjoyment. Objective gameplay metrics, including match results and movement heatmaps, were also collected. Results: The self-play model was consistently perceived as the most difficult. In contrast, models trained against deterministic opponents with difficulty scaling were rated as more natural and more enjoyable. These results indicate that higher competitive performance does not necessarily lead to better player experience. Limitations include the small sample size and evaluation in a single scenario. Future work involves testing additional RL algorithms and extending the environment to more complex or varied game contexts.

Palavras-chave: Reinforcement Learning, self-play, Curriculum Learning, Video Games, User Experience

Referências

Albrecht, S. V., Christianos, F., e Schäfer, L. (2024). Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press, Cambridge, MA, USA.

Almeida, P., Carvalho, V., e Simões, A. (2024). Reinforcement learning as an approach to train multiplayer first-person shooter game agents. Technologies, 12(3):34.

Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., e Mordatch, I. (2020). Emergent tool use from multi-agent autocurricula. arXiv:1909.07528.

Cohen, A., Teng, E., Berges, V.-P., Dong, R.-P., Henry, H., Mattar, M., Zook, A., e Ganguly, S. (2022). On the use and misuse of absorbing states in multi-agent reinforcement learning. arXiv:2111.05992.

DiGiovanni, A. e Zell, E. C. (2021). Survey of self-play in reinforcement learning.

Haddad, G., Julia, R., e Faria, M. (2021). Técnicas de aprendizado por reforço aplicadas em jogos eletrônicos na plataforma geral do unity. In Anais do XVIII Encontro Nacional de Inteligência Artificial e Computacional, pages 571–582, Porto Alegre, RS, Brasil. SBC.

Huh, D. e Mohapatra, P. (2024). Multi-agent reinforcement learning: A comprehensive survey. arXiv:2312.10256.

Huynh, N.-M., Cao, H.-G., e Wu, I.-C. (2025). Multi-agent training for pommerman: Curriculum learning and population-based self-play approach. arXiv:2407.00662.

Li, Z., Ji, Q., Ling, X., e Liu, Q. (2025). A comprehensive review of multiagent reinforcement learning in video games. IEEE Transactions on Games, 17(4):873–892.

Milani, S., Juliani, A., Momennejad, I., Georgescu, R., Rzepecki, J., Shaw, A., Costello, G., Fang, F., Devlin, S., e Hofmann, K. (2023). Navigates like me: Understanding how people evaluate human-like ai in video games. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Proc. CHI 2023), pages 1–18, Hamburg, Germany. ACM.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., e Klimov, O. (2017). Proximal policy optimization algorithms.

Servat, A. e Mohamadi, H. S. (2023). Immersive game worlds: Using deep reinforcement learning for lifelike non-player characters. In Proceedings of the 2023 International Serious Games Symposium (Proc. ISGS 2023), pages 1–5, Tehran, Iran. IEEE.

Sutton, R. S. e Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 2 edition.

Yannakakis, G. N. e Togelius, J. (2025). Artificial Intelligence and Games. Springer Nature, Cham, Switzerland.