Reinforcement Learning Applied to Very Small Size Soccer Decision-Making, Trajectory Planning and Control In Penalty Kicks

• Thayna Pires Baldão ITA
• Marcos R. O. A. Maximo ITA
• Takashi Yoneyama ITA

Abstract

In this work, we propose a Reinforcement Learning (RL) training methodology using Proximal Policy Optimization (PPO) and Curriculum Learning (CL) to make simulated Very Small Size Soccer robots learn to convert penalty kicks against two types of opponents: the “line follow” goalkeeper, which defends the goal by following the goal line, and the offensive goalkeeper, which uses the univector field navigation technique to attack the ball. The RL agents trained using this methodology demonstrated sophisticated penalty kick skills with high conversion rates. The best-evaluated RL agent achieved a conversion rate of over 90% against the “line follow” goalkeeper and of over 71% against the offensive goalkeeper. Comparisons with the univector agent showed that the RL agents significantly outperformed it, with conversion rates 78.71% higher against the “line follow” goalkeeper and 1997.66% higher against the offensive goalkeeper.

Keywords: Training, Knowledge engineering, Trajectory planning, Navigation, Neural networks, Decision making, Reinforcement learning, Robots, Optimization, Sports, Deep Reinforcement Learning, Proximal Policy Optimization, Curriculum Learning, Robotics, Robot Soccer