Deep Reinforcement Learning for Mapless Navigation of a Hybrid Aerial Underwater Vehicle

• Ricardo B. Grando FURG
• Paulo L. J. Drews-Jr FURG

DOI: https://doi.org/10.5753/ctd.2022.222936

Abstract

The search for the development of new technologies drives great challenges. An example of this refers to the development of tasks related to hybrid mobile robots. This work presents an approach based on deep reinforcement learning (Deep-RL) for autonomous navigation of a specific type of hybrid mobile robot: a Hybrid Unmanned Aerial Underwater Vehicle (HUAUV). OThe proposed approach uses only information from distance sensors and information related to the location of the vehicle to perform navigation. Results of our approach show that it is possible to perform navigation without a map from start to finish, without the need to use any type of manual operation, only using Deep- RL-based agents. For that, the navigation of the trained agents is compared with the navigation without a map performed by an algorithm BUG2, a modern version of a standard algorithm without learning for the problem. The proposed methods are based on two state-of-the-art approaches to map-less navigation of land robots: Deep Deterministic Policy Gradient (DDPG) and Soft Actor Critic (SAC).

Keywords: Deep-RL, Mapless Navigation, Autonomous Robot, Hybrid Vehicle

References

Cerqueira, R., Trocoli, T., Neves, G., Oliveira, L., Joyeux, S., Albiez, J., and Center, R. I. (2016). Custom shader and 3d rendering for computationally efficient sonar simulation. In SIBGRAPI.

Drews, P. L., Neto, A. A., and Campos, M. F. (2014). Hybrid unmanned aerial underwater vehicle: Modeling and simulation. In IEEE/RSJ IROS, pages 4637–4642.

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290.

Horn, A. C., Pinheiro, P. M., Grando, R. B., da Silva, C. B., Neto, A. A., and Drews-Jr, P. L. (2020). A novel concept for hybrid unmanned aerial underwater vehicles focused on aquatic performance. In IEEE LARS/SBR, pages 1–6.

Horn, A. C., Pinheiro, P. M., Silva, C. B., Neto, A. A., and Drews-Jr, P. L. (2019). A study on configuration of propellers for multirotor-like hybrid aerial-aquatic vehicles. In ICAR, pages 173–178.

Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274.

Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.

Marino, R., Mastrogiovanni, F., Sgorbissa, A., and Zaccaria, R. (2016). A minimalistic quadrotor navigation strategy for indoor multi-floor scenarios. In Intelligent Autonomous Systems 13, pages 1561–1570. Springer.

Mercado, D., Maia, M., and Diez, F. J. (2019). Aerial-underwater systems, a new paradigm in unmanned vehicles. Journal of Intelligent & Robotic Systems, 95(1):229–238.

Srinivas, A., Laskin, M., and Abbeel, P. (2020). Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136.

Tai, L., Paolo, G., and Liu, M. (2017). Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. In IEEE/RSJ IROS, pages 31–36.

Zhu, K. and Zhang, T. (2021). Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology, 26(5):674–691.