A DRL Approach for Mapless Planar Pushing of Arbitrary Objects in Cluttered Environments

• Gabriel Santos Luz UFMG
• Douglas G. Macharet UFMG

DOI: https://doi.org/10.5753/sbrlars_estendido.2025.248248

Resumo

Pushing is a fundamental yet challenging primitive in robotics, especially in cluttered and constrained environments. This dissertation proposes a novel two-level approach combining a low-level Deep Reinforcement Learning (DRL) policy and a high-level navigator to transport objects through narrow passages. The DRL policy ensures the object stays within a tight capsule, enabling integration with classical planners. Experiments show that this method reliably pushes irregular objects through spaces as narrow as twice their diameter, outperforming unconstrained methods, and succeeds in complex, mapless environments with dead ends and tight corridors.

Palavras-chave: Planar Pushing, Mobile Robotics, Robotic Manipulation, Reinforcement Learning, Deep Learning

Referências

Bauza, M., Alet, F., Lin, Y.-C., Lozano-Pérez, T., Kaelbling, L. P., Isola, P., and Rodriguez, A. (2019). Omnipush: accurate, diverse, real-world dataset of pushing dynamics with rgb-d video. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4265–4272. IEEE.

Bauza, M., Hogan, F. R., and Rodriguez, A. (2018). A data-efficient approach to precise and controlled pushing. In Conference on Robot Learning, pages 336–345. PMLR.

Bauza, M. and Rodriguez, A. (2017). A probabilistic data-driven model for planar pushing. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3008–3015. IEEE.

Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. (2023). Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668.

Cho, Y., Han, J., Cho, Y., and Kim, B. (2024). Corn: Contact-based object representation for nonprehensile manipulation of general unseen objects. arXiv preprint arXiv:2403.10760.

Clavera, I., Held, D., and Abbeel, P. (2017). Policy transfer via modularity and reward guiding. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1537–1544. IEEE.

Cong, L., Liang, H., Ruppel, P., Shi, Y., Görner, M., Hendrich, N., and Zhang, J. (2022). Reinforcement learning with vision-proprioception model for robot planar pushing. Frontiers in Neurorobotics, 16:829437.

Dengler, N., Ferrandis, J. D. A., Moura, J., Vijayakumar, S., and Bennewitz, M. (2024). Learning goal-directed object pushing in cluttered scenes with location-based attention. arXiv preprint arXiv:2403.17667.

Dengler, N., Großklaus, D., and Bennewitz, M. (2022). Learning goal-oriented non-prehensile pushing in cluttered scenes. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1116–1122. IEEE.

Dogar, M. R. and Srinivasa, S. S. (2010). Push-grasping with dexterous hands: Mechanics and a method. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2123–2130. IEEE.

Eoh, G. (2023). Deep-reinforcement-learning-based object transportation using task space decomposition. Sensors, 23(10):4807.

Ferrandis, J. D. A., Moura, J., and Vijayakumar, S. (2023). Nonprehensile planar manipulation through reinforcement learning with multimodal categorical exploration. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5606–5613. IEEE.

Ferrandis, J. D. A., Moura, J., and Vijayakumar, S. (2024). Learning visuotactile estimation and control for non-prehensile manipulation under occlusions. arXiv preprint arXiv:2412.13157.

Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., and Levine, S. (2018). Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. ArXiv, abs/1806.10293.

Kavraki, L. E., Svestka, P., Latombe, J.-C., and Overmars, M. H. (1996). Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE transactions on Robotics and Automation, 12(4):566–580.

King, J. E., Klingensmith, M., Dellin, C. M., Dogar, M. R., Velagapudi, P., Pollard, N. S., and Srinivasa, S. S. (2013). Pregrasp manipulation as trajectory optimization. In Robotics: Science and Systems. Berlin.

Kloss, A., Bauza, M., Wu, J., Tenenbaum, J. B., Rodriguez, A., and Bohg, J. (2020). Accurate vision-based manipulation through contact reasoning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6738–6744. IEEE.

LaValle, S. (1998). Rapidly-exploring random trees: A new tool for path planning. Research Report 9811.

Lee, G., Lozano-Pérez, T., and Kaelbling, L. P. (2015). Hierarchical planning for multi-contact non-prehensile manipulation. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 264–271. IEEE.

Lin, T., Sachdev, K., Fan, L., Malik, J., and Zhu, Y. (2025). Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids. arXiv preprint arXiv:2502.20396.

Liu, M., Zhu, M., and Zhang, W. (2022). Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08299.

Lynch, K. M. (1996). Nonprehensile robotic manipulation: Controllability and planning. Carnegie Mellon University.

Mandadi, V. R., Saha, K., Guhathakurta, D., Qureshi, M. N., Agarwal, A., Sen, B., Das, D., Bhowmick, B., Singh, A., and Krishna, M. (2023). Disentangling planning and control for non-prehensile tabletop manipulation. In 2023 IEEE 19th International Conference on Automation Science and Engineering (CASE), pages 1–6. IEEE.

Mason, M. T. (1986). Mechanics and planning of manipulator pushing operations. The International Journal of Robotics Research, 5(3):53–71.

Ng, A. Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer.

OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., teusz Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N. A., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L. M. (2019). Solving rubik’s cube with a robot hand. ArXiv, abs/1910.07113.

Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803–3810. IEEE.

Pertsch, K., Stachowicz, K., Ichter, B., Driess, D., Nair, S., Vuong, Q., Mees, O., Finn, C., and Levine, S. (2025). Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747.

Salganicoff, M., Metta, G., Oddera, A., and Sandini, G. (1993). A vision-based learning method for pushing manipulation, volume 54. University of Pennsylvania.

Wang, S., Sun, L., Zha, F., Guo, W., and Wang, P. (2023). Learning adaptive reaching and pushing skills using contact information. Frontiers in Neurorobotics, 17:1271607.

Yang, M., Lin, Y., Church, A., Lloyd, J., Zhang, D., Barton, D. A., and Lepora, N. F. (2023). Sim-to-real model-based and model-free deep reinforcement learning for tactile pushing. IEEE Robotics and Automation Letters, 8(9):5480–5487.

Yu, K.-T., Bauza, M., Fazeli, N., and Rodriguez, A. (2016). More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 30–37. IEEE.

Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., and Funkhouser, T. (2018). Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4238–4245. IEEE.