Sim-to-Real in Reinforcement Learning for Everyone

• Juliano Vacaro HP inc
• Guilherme Marques HP inc
• Bruna Oliveira HP inc
• Gabriel Paz HP inc
• Thomas Paula HP inc
• Wagston Staehler HP inc
• David Murphy HP inc

Resumo

In reinforcement learning (RL), it remains a challenge to have a robotic agent perform a task in the real world for which it was trained in simulation. In this paper, we present our work training a low-cost robotic arm in simulation to move towards a predefined target in space, represented by a red ball in an RGB image, and transferring the capability to the real arm. We exercised the entire end-to-end flow including the 3D modeling of the arm, training of a state-of-the-art RL policy in simulation with multiple actors in a distributed fashion, domain randomization in order to close the sim-to-real gap, and finally the execution of the trained model in the real robot. We also implemented a mechanism to edit the image captured from the camera before sending it to the model for inference, which allowed us to automate reward computation in the physical world. Our work highlights important challenges of training RL agents and moving them to the real world, validating important aspects shown by other works as well as detailing steps not explained by some of them (e.g. how to compute the reward in the real world). The conducted experiments show the improvements observed as the techniques were added to the final solution.

Palavras-chave: Manipulators, Training, Task analysis, Learning (artificial intelligence), Solid modeling, Cameras