Deep Reinforcement Learning Applied to IEEE’s Very Small Size Soccer in Cooperative Attack Strategy

• Thiago Filipe de Medeiros ITA
• Takashi Yoneyama ITA
• Marcos R. O. A. Máximo ITA

Resumo

In this work, we use a virtual learning framework for the IEEE’s Very Small Size Soccer (VSSS) competition based on Deep Reinforcement Learning (DRL) to train and improve the behavior of agents within the game strategy. The agent was trained on a scenario where a pair of robots must cooperate to score a goal in the adversary field. To achieve that, the concept of Self-Play was used, where the agent learns to cooperate with a copy of itself. Furthermore, the Proximal Policy optimization (PPO) technique was used during the training process. Finally, the trained agent’s performance was evaluated and compared with agents following handmade heuristic behaviors in two scenarios: (1) empty goal (in which the DRL agent was trained on) and (2) against goalie (not seen by the agent during train). As a result, the DRL agent’s outperformed the heuristic behavior agents in both scenarios when comparing success rate: (1) 99.18% against 81.50% and (2) 32.94% against 28.60%. The contribution of this work is the usage of cooperative Self-Play in an offensive move. To the best of our knowledge, this is the first work to succeed in combining Self-Play and the PPO technique in IEEE’s VSSS.