Two Level Control of Non-Player Characters for Navigation in 3D Games Scenes: A Deep Reinforcement Learning Approach

• Gilzamir Gomes UFC
• Creto A. Vidal UFC
• Joaquim B. Cavalcante-Neto UFC
• Yuri L. B. Nogueira UFC

Resumo

This paper presents a deep reinforcement learning approach for navigation problem coupled with traditional animation processes in games. Deep Reinforcement Learning (DRL) is a promising approach for this problem. So, we design a NonPlayer Character (NPC) as an autonomous agent guided by a neural controller. Our approach works with the control of virtual game characters at two different levels of abstraction. A neural controller produces high-level actions, which are performed by both a character controller and an animation controller. We test our approach on a three-dimensional game scene. We find that our approach achieves promising results in navigation of animated characters in a game scene.

Palavras-chave: non-player characters, reinforcement learning, navigation problem

Referências

L. Lidén, “Strategic and tactical reasoning with waypoints,” in AI Game Programming Wisdom, S. Rabin, Ed. Hingham, MA, USA: Charles River Media, 2002, pp. 211–220.

E. Alonso, M. Peter, D. Goumard, and J. Romoff, “Deep reinforcement learning for navigation in aaa video games,” arXiv preprint arXiv:2011.04764, 2020.

P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu et al., “Learning to navigate in complex environments,” arXiv preprint arXiv:1611.03673, 2016.

S. Phon-Amnuaisuk, “Learning chasing behaviours of non-player characters in games using sarsa,” in Applications of Evolutionary Computation, C. Di Chio, S. Cagnoni, C. Cotta, M. Ebner, A. Ekárt, A. I. Esparcia-Alcázar, J. J. Merelo, F. Neri, M. Preuss, H. Richter, J. Togelius, and G. N. Yannakakis, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 133–142.

T. Barron, M. Whitehead, and A. Yeung, “Deep reinforcement learning in a 3-d blockworld environment,” Deep Reinforcement Learning: Frontiers and Challenges, IJCAI, vol. 2016, p. 16, 2016.

F. G. Glavin and M. G. Madden, “Learning to shoot in first person shooter games by stabilizing actions and clustering rewards for reinforcement learning,” in 2015 IEEE Conference on Computational Intelligence and Games (CIG), Aug 2015, pp. 344–351.

J. Beck, K. Ciosek, S. Devlin, S. Tschiatschek, C. Zhang, and K. Hofmann, “Amrl: Aggregated memory for reinforcement learning,” in International Conference on Learning Representations, 2019.

A. Dobrovsky, U. Borghoff, and M. Hofmann, “Applying and augmenting deep reinforcement learning in serious games through interaction,” Periodica Polytechnica Electrical Engineering and Computer Science, vol. 61, no. 2, pp. 198–208, 2017. [Online]. Available: https://pp.bme.hu/eecs/article/view/10313

A. Juliani, V.-P. Berges, E. Vckay, Y. Gao, H. Henry, M. Mattar, and D. Lange, “Unity: A general platform for intelligent agents,” arXiv preprint arXiv:1809.02627, 2020.

G. Gomes, C. A. Vidal, J. B. Cavalcante-Neto, and Y. L. Nogueira, “Ai4u: A tool for game reinforcement learning experiments,” in 2020 19th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames). IEEE, 2020, pp. 19–28.

G. Gomes, C. A. Vidal, J. B. Cavalcante Neto, and Y. L. B. Nogueira, “An emotional virtual character: A deep learning approach with reinforcement learning,” in 2019 21st Symposium on Virtual and Augmented Reality (SVR), Oct 2019, pp. 223–231.

P. Mirowski, “Learning to navigate,” in 1st International Workshop on Multimodal Understanding and Learning for Embodied Applications, 2019, pp. 25–25.

E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames,” in International Conference on Learning Representations, 2019.

S. Bansal, V. Tolani, S. Gupta, J. Malik, and C. Tomlin, “Combining optimal control and learning for visual navigation in novel environments,” in Proceedings of the Conference on Robot Learning, ser. Proceedings of Machine Learning Research, L. P. Kaelbling, D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 30 Oct–01 Nov 2020, pp. 420–429.

B. Eysenbach, R. R. Salakhutdinov, and S. Levine, “Search on the replay buffer: Bridging planning and reinforcement learning,” in Advances in Neural Information Processing Systems, 2019, pp. 15 246–15 257.

X. Meng, N. Ratliff, Y. Xiang, and D. Fox, “Scaling local control to large-scale topological navigation,” arXiv preprint arXiv:1909.12329, 2019.

S. Hochreiter and J. Schmidhuber, “Lstm can solve hard long time lag problems,” in Advances in neural information processing systems, 1997, pp. 473–479.

E. Parisotto and R. Salakhutdinov, “Neural map: Structured memory for deep reinforcement learning,” arXiv preprint arXiv:1702.08360, 2017.

E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf, “Egomap: Projective mapping and structured egocentric memory for deep rl,” in European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 2020.

M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience replay,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 5055–5065.

D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devin, B. Eysenbach, and S. Levine, “Learning to reach goals without reinforcement learning,” arXiv preprint arXiv:1912.06088, 2019.

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller, “Playing atari with deep reinforcement learning,” CoRR, vol. abs/1312.5602, 2013.

H. Ide and T. Kurita, “Improvement of learning for cnn with relu activation by sparse regularization,” 05 2017, pp. 2684–2691.