Improved Defensive Strategy in RoboCup Small Size League using Deep Reinforcement Learning

• Adrisson R. Samersla ITA
• Marcos R. O. A. Maximo ITA

Abstract

The Small Size League (SSL) is one of the old-est leagues in RoboCup, an international scientific initiative whose objective is to advance the state-of-the-art in robotics. The matches are highly dynamic and competitive, an excellent environment to exercise and extend the latest decision-making techniques. In this work, we focus on the planning problem of the defense strategy in the SSL context. Our contribution lies in proposing a method for learning such behaviors using Deep Reinforcement Learning (DRL) algorithms. Through this technique, we intend to apply a more automated process than manually adjusting the heuristics used to handcraft such skills in code. For this purpose, the RoboCup Small Size League standard simulator grSim was adapted in order to implement a synchronous mode. Furthermore, we integrated this new simulator with ITAndroids team's code base and with the DRL algorithms provided by the Stable Baselines 3 library. With this setup, we successfully trained an agent using Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), the state-of-the-art for continuous control reinforcement learning algorithms. Moreover, we demonstrate that the resulting policies outperform the hand-coded counterparts, showing potential to improve the team's overall defense strategy.

Keywords: Codes, Deep reinforcement learning, Libraries, Planning, Reliability, Robots, Standards, Optimization, Sports, Overfitting, RoboCup, Small size league