SLAM Visual Em Ambientes Dinâmicos Usando Segmentação Panóptica
Abstract
The majority of visual SLAM systems are not robust in dynamic scenarios. The ones that deal with dynamic content in the scenes usually rely on deep learning-based methods to detect and filter dynamic objects. However, these methods cannot deal with unknown objects. This work presents Panoptic-SLAM, a visual SLAM system robust to dynamic environments, even in the presence of unknown objects. It uses Panoptic Segmentation to filter dynamic objects from the scene during the state estimation process. The proposed methodology is based on ORB-SLAM3 [Campos et al. 2021], a state-of-the-art SLAM system for static environments. The implementation was tested using real-world datasets and compared with several systems from the literature, including DynaSLAM, DS-SLAM and SaD-SLAM and PVO.
References
Bescos, B., Fácil, J. M., Civera, J., and Neira, J. (2018). Dynaslam: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robotics and Automation Letters, 3(4):4076–4083.
Bolya, D., Zhou, C., Xiao, F., and Lee, Y. J. (2019). Yolact: Real-time instance segmentation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9156–9165.
Campos, C., Elvira, R., Rodríguez, J. J. G., Montiel, J. M. M., and Tardós, J. D. (2021). Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874–1890.
Engel, J., Koltun, V., and Cremers, D. (2018). Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):611–625.
Engel, J., Schöps, T., and Cremers, D. (2014). Lsd-slam: Large-scale direct monocular slam. In Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II, pages 834–849.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969.
Ji, T., Wang, C., and Xie, L. (2021). Towards real-time semantic rgb-d slam in dynamic environments. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11175–11181.
Kirillov, A., Girshick, R., He, K., and Dollar, P. (2019a). Panoptic feature pyramid networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6392–6401.
Kirillov, A., He, K., Girshick, R., Rother, C., and Dollár, P. (2019b). Panoptic segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9396–9405.
Li, G. and Chen, S. (2022). Visual slam in dynamic scenes based on object tracking and static points detection. Journal of Intelligent & Robotic Systems, 104(33).
Lin, T., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
Liu, Y. and Miura, J. (2021). Rds-slam: Real-time dynamic slam using semantic segmentation methods. IEEE Access, 9:23772–23785.
Mur-Artal, R., Montiel, J. M. M., and Tardós, J. D. (2015). Orb-slam: A versatile and accurate monocular slam system. IEEE Transactions on Robotics, 31(5):1147–1163.
Mur-Artal, R. and Tardós, J. D. (2017). Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 33(5):1255–1262.
Palazzolo, E., Behley, J., Lottes, P., Giguère, P., and Stachniss, C. (2019). Refusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7855–7862.
Redmon, J. and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767.
Soares, J. C. V., Gattass, M., and Meggiolaro, M. A. (2021). Crowd-slam: Visual slam towards crowded environments using object detection. Journal of Intelligent & Robotic Systems, 102(2).
Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cremers, D. (2012). A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 573–580.
Teed, Z. and Deng, J. (2021). Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. In Neural Information Processing Systems.
Vincent, J., Labb’e, M., Lauzon, J., Grondin, F., Comtois-Rivet, P., and Michaud, F. (2020). Dynamic object tracking and masking for visual slam. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4974–4979.
Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y., and Girshick, R. (2019). Detectron2. https://github.com/facebookresearch/detectron2.
Yang, S., Fan, G., Bai, L., Zhao, C., and Li, D. (2020). Sgc-vslam: A semantic and geometric constraints vslam for dynamic indoor environments. Sensors, 20(8).
Ye, W., Lan, X., Chen, S., Ming, Y., rong Yu, X., Bao, H., Cui, Z., and Zhang, G. (2022). Pvo: Panoptic visual odometry. ArXiv, abs/2207.01610.
Yu, C., Liu, Z., Liu, X.-J., Xie, F., Yang, Y., Wei, Q., and Fei, Q. (2018). Ds-slam: A semantic visual slam towards dynamic environments. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1168–1174.
Yuan, X. and Chen, S. (2020). Sad-slam: A visual slam based on semantic and depth information. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4930–4935.
Yuan, Z., Xu, K., Zhou, X., Deng, B., and Ma, Y. (2021). SVG-Loop : Semantic – Visual – Geometric Information-Based Loop Closure Detection.
Zhang, J., Gao, M., He, Z., and Yang, Y. (2022). Dcs-slam: A semantic slam with moving cluster towards dynamic environments. In 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 1923–1928.
Zhu, H., Yao, C., Zhu, Z., Liu, Z., and Jia, Z. (2022). Fusing panoptic segmentation and geometry information for robust visual slam in dynamic environments. In IEEE 18th International Conference on Automation Science and Engineering.
Bolya, D., Zhou, C., Xiao, F., and Lee, Y. J. (2019). Yolact: Real-time instance segmentation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9156–9165.
Campos, C., Elvira, R., Rodríguez, J. J. G., Montiel, J. M. M., and Tardós, J. D. (2021). Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874–1890.
Engel, J., Koltun, V., and Cremers, D. (2018). Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):611–625.
Engel, J., Schöps, T., and Cremers, D. (2014). Lsd-slam: Large-scale direct monocular slam. In Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II, pages 834–849.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969.
Ji, T., Wang, C., and Xie, L. (2021). Towards real-time semantic rgb-d slam in dynamic environments. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11175–11181.
Kirillov, A., Girshick, R., He, K., and Dollar, P. (2019a). Panoptic feature pyramid networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6392–6401.
Kirillov, A., He, K., Girshick, R., Rother, C., and Dollár, P. (2019b). Panoptic segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9396–9405.
Li, G. and Chen, S. (2022). Visual slam in dynamic scenes based on object tracking and static points detection. Journal of Intelligent & Robotic Systems, 104(33).
Lin, T., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
Liu, Y. and Miura, J. (2021). Rds-slam: Real-time dynamic slam using semantic segmentation methods. IEEE Access, 9:23772–23785.
Mur-Artal, R., Montiel, J. M. M., and Tardós, J. D. (2015). Orb-slam: A versatile and accurate monocular slam system. IEEE Transactions on Robotics, 31(5):1147–1163.
Mur-Artal, R. and Tardós, J. D. (2017). Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 33(5):1255–1262.
Palazzolo, E., Behley, J., Lottes, P., Giguère, P., and Stachniss, C. (2019). Refusion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7855–7862.
Redmon, J. and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767.
Soares, J. C. V., Gattass, M., and Meggiolaro, M. A. (2021). Crowd-slam: Visual slam towards crowded environments using object detection. Journal of Intelligent & Robotic Systems, 102(2).
Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cremers, D. (2012). A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 573–580.
Teed, Z. and Deng, J. (2021). Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. In Neural Information Processing Systems.
Vincent, J., Labb’e, M., Lauzon, J., Grondin, F., Comtois-Rivet, P., and Michaud, F. (2020). Dynamic object tracking and masking for visual slam. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4974–4979.
Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y., and Girshick, R. (2019). Detectron2. https://github.com/facebookresearch/detectron2.
Yang, S., Fan, G., Bai, L., Zhao, C., and Li, D. (2020). Sgc-vslam: A semantic and geometric constraints vslam for dynamic indoor environments. Sensors, 20(8).
Ye, W., Lan, X., Chen, S., Ming, Y., rong Yu, X., Bao, H., Cui, Z., and Zhang, G. (2022). Pvo: Panoptic visual odometry. ArXiv, abs/2207.01610.
Yu, C., Liu, Z., Liu, X.-J., Xie, F., Yang, Y., Wei, Q., and Fei, Q. (2018). Ds-slam: A semantic visual slam towards dynamic environments. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1168–1174.
Yuan, X. and Chen, S. (2020). Sad-slam: A visual slam based on semantic and depth information. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4930–4935.
Yuan, Z., Xu, K., Zhou, X., Deng, B., and Ma, Y. (2021). SVG-Loop : Semantic – Visual – Geometric Information-Based Loop Closure Detection.
Zhang, J., Gao, M., He, Z., and Yang, Y. (2022). Dcs-slam: A semantic slam with moving cluster towards dynamic environments. In 2022 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 1923–1928.
Zhu, H., Yao, C., Zhu, Z., Liu, Z., and Jia, Z. (2022). Fusing panoptic segmentation and geometry information for robust visual slam in dynamic environments. In IEEE 18th International Conference on Automation Science and Engineering.
Published
2023-10-09
How to Cite
ABATI, Gabriel F.; SOARES, João Carlos V.; MEGGIOLARO, Marco Antonio.
SLAM Visual Em Ambientes Dinâmicos Usando Segmentação Panóptica. In: GRADUATE WORKS CONTEST IN ROBOTICS - CTDR (MSC) - BRAZILIAN SYMPOSIUM OF ROBOTICS & LATIN AMERICAN ROBOTICS SYMPOSIUM (SBR/LARS), 15. , 2023, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 61-72.
DOI: https://doi.org/10.5753/sbrlars_estendido.2023.235116.
