Investigating Learning Methods and Environment Representation in the Construction of Player Agents: Application on FIFA Game

• Matheus Prado Prandini Faria UFU http://orcid.org/0000-0003-1468-9243
• Rita Maria Silva Julia UFU
• Lídia Bononi Paiva Tomaz IFTM

DOI: https://doi.org/10.5753/sbgames_estendido.2021.19744

Resumo

The objective behind this study is to investigate Machine Learning (ML) techniques combined with methods from Computer Vision (CV) for state representation by images, to produce agents capable of solving problems, in real time, in environments with complex properties. Such difficulties require agents to be highly efficient in their learning (and, consequently, decision-making) and environmental perception processes, without which they will not be successful. The digital game FIFA - soccer simulator - is used as a case study because it represents a realistic and challenging environment. The ML techniques are investigated in the context of the Deep Learning (DL) approach provided by Convolutional Neural Networks (CNNs), being: imitation learning, used here with the purpose of endowing the agent with the ability to solve problems in a way closer to human; by deep reinforcement, in which the agent is trained in an attempt to autonomously abstract an optimal decision-making policy. Regarding the environmental perception, the following state representations approaches are investigated in this study: raw images - with and without color information - and through Object Detection Techniques (ODT). In order to further improve the performance of the agents produced, genetic algorithm techniques are explored to automatically define a CNN architecture to be used as the player agents decision-making module. In addition to corroborating the excellent results that DL combined with CV has been producing in the context of ML (particularly in games), the present work shows the great potential of the application of ODT in the process of enhancing the environmental perception, which counts as a relevant counterpart to the fact that ODT demands computational procedures with a higher cost in relation to the representations based on raw images.

Palavras-chave: Deep Learning, Imitation Learning, Deep Reinforcement Learning, Object Detection Techniques, Genetic Algorithms

Referências

Y. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.

I. Goodfellow, Y. Bengio, and A. Courville, "Deep Learning," MIT Press, 2016, book in preparation for MIT Press.

W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, “A survey of deep neural network architectures and their applications,” Neurocomputing, vol. 234, pp. 11–26, 2017. [Online], doi: 10.1016/j.neucom.2016.12.038.

L. B. P. Tomaz, “Adaba: a new alpha-beta distribution approach-application to the checkers game domain,” Ph.D. dissertation, Universidade Federal de Uberlândia, 2019.

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in Computer Vision–ECCV, 2016, pp. 21–37.

V. Mnih, K. Kavukcuoglu, D. Silver, and A. A. Rusu, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.

K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” in IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 26–38, 2017.

R. S. Sutton and A. G. Barto, "Introduction to Reinforcement Learning," 1st ed. Cambridge, MA, USA:MIT Press, 1998.

M. Bain and C. Sammut, “A framework for behavioural cloning,” Machine Intelligence 15, 1995.

Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation:Learning to imitate behaviors from raw video via context translation,” in IEEE International Conference on Robotics and Automation (ICRA), pp. 1118–1125, 2018.

A. Hussein, E. Elyan, M. M. Gaber, and C. Jayne, “Deep imitation learning for 3d navigation tasks,” Neural Computing and Applications, vol. 29, no. 7, pp. 389–404, April 2018.

M. P. P. Faria, R. M. S. Julia, and L. B. P. Tomaz, “A deep reinforcement learn-based fifa agent with naive state representations and portable connection interfaces,” in International Florida Artificial Intelligence Research Society Conference (FLAIRS), 2019, pp. 282–285.

M. P. P. Faria, R. M. S. Julia, and L. B. P. Tomaz, "Improving FIFA player agents decision-making architectures based on convolutional neural networks through evolutionary techniques,” Intelligent Systems - 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20-23, 2020, Proceedings, Part I, ser. Lecture Notes in Computer Science, R. Cerri and R. C. Prati, Eds., vol.12319. Springer, 2020, pp. 371–386.

M. P. P. Faria, R. M. S. Julia, and L. B. P. Tomaz, "Improving fifa free kicks player agent performance through object detection techniques,” in Proceedings of SBGames 2020 - XIX Simpósio Brasileiro de Jogos e Entretenimento Digital.

M. P. P. Faria, R. M. S. Julia, and L. B. P. Tomaz, "Deep active imitation learning in fifa free-kicks player platforms based on raw image and object detection state representations,” in 31st International Conference on Tools with Artificial Intelligence, 2019.

M. P. P. Faria, R. M. S. Julia, and L. B. P. Tomaz, "Evaluating the performance of the deep active imitation learning algorithm in the dynamic environment of fifa player agents,” in IEEE International Conference on Machine Learning and Applications, 2019.