Dense Recognition for Understanding Ambiguities in Urban Environments of Autonomous Vehicles

• Wilbur N. Chiuyari-Veramendi USP
• Fernando S. Osório USP

DOI: https://doi.org/10.5753/sibgrapi.est.2025.38298

Abstract

In autonomous navigation of mobile robots and vehicles, dense scene understanding—through semantic segmentation and depth estimation—is essential to ensure safety and adaptability in complex urban environments. Despite recent advances in computer vision, such as vision transformers and foundational models, there are still few studies that unify dense recognition applied to mobile robots and autonomous vehicles. This work proposes a lightweight, integrated architecture that exploits the complementarity between these two tasks by combining representations extracted from foundational models like DINOv2 and Depth Anything. Semantic-geometric integration strategies, including depth map concatenation, weighted spatial attention, and cross-attention, are investigated to enhance segmentation robustness in ambiguous situations, such as reflections, occlusions, and unreal objects. The approach is evaluated on an embedded platform (Jetson AGX Orin), considering quantitative metrics (mIoU, AbsRel, FPS) and qualitative assessments, focusing on computational feasibility. Expected results indicate that, even with a frozen encoder, training limited to the heads combined with efficient strategies can achieve a relevant balance between semantic performance, geometric accuracy, and real-time resource usage.

References

J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.

D. Eigen and R. Fergus, “Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2650–2658.

X. Yu, Y. Zuo, Z. Wang, X. Zhang, J. Zhao, Y. Yang, L. Jiao, R. Peng, X. Wang, J. Zhang, K. Zhang, F. Liu, R. Alcover-Couso, J. C. SanMiguel, M. Escudero-Viñolo, H. Tian, K. Matsui, T.Wang, F. Adan, Z. Gao, X. He, Q. Bouniot, H. Moghaddam, S. N. Rai, F. Cermelli, C. Masone, A. Pilzer, E. Ricci, A. Bursuc, A. Solin, M. Trapp, R. Li, A. Yao, W. Chen, I. Simpson, N. D. F. Campbell, and G. Franchi, “The robust semantic segmentation uncv2023 challenge results,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, October 2023, pp. 4618–4628.

J. Spencer, C. S. Qian, C. Russell, S. Hadfield, E. Graf, W. Adams, A. J. Schofield, J. H. Elder, R. Bowden, H. Cong, S. Mattoccia, M. Poggi, Z. K. Suri, Y. Tang, F. Tosi, H. Wang, Y. Zhang, Y. Zhang, and C. Zhao, “The monocular depth estimation challenge,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, January 2023, pp. 623–632.

C. Zhao, Q. Sun, C. Zhang, Y. Tang, and F. Qian, “Monocular depth estimation based on deep learning: An overview,” Science China Technological Sciences, vol. 63, no. 9, pp. 1612–1627, 2020.

T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1851–1858.

A. Pilzer, S. Lathuiliere, N. Sebe, and E. Ricci, “Refine and distill: Exploiting cycle-inconsistency and knowledge distillation for unsupervised monocular depth estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9768–9777.

B. Cheng, A. Schwing, and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” Advances in neural information processing systems, vol. 34, pp. 17 864–17 875, 2021.

Y. Zhao, L. Wang, X. Yun, C. Chai, Z. Liu, W. Fan, X. Luo, Y. Liu, and X. Qu, “Enhanced scene understanding and situation awareness for autonomous vehicles based on semantic segmentation,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026.

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 10 371–10 381.

D. R. Bruno, R. Berri, F. Barbosa, and F. S. Osorio, “Carina project: Visual perception systems applied for autonomous vehicles and advanced driver assistance systems (adas),” IEEE Access, 2023.

M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Advances in neural information processing systems, vol. 33, pp. 9912–9924, 2020.

M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023.

Q. Tang, F. Liu, T. Zhang, J. Jiang, Y. Zhang, B. Zhu, and X. Tang, “Compensating for local ambiguity with encoder-decoder in urban scene segmentation,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 10, pp. 19 224–19 235, 2022.

X. Zhou, M. Liu, B. Luka Zagar, E. Yurtsever, and A. C. Knoll, “Vision language models in autonomous driving and intelligent transportation systems,” arXiv e-prints, pp. arXiv–2310, 2023.

X. Shan and C. Zhang, “Robustness of segment anything model (sam) for autonomous driving in adverse weather conditions,” arXiv preprint arXiv:2306.13290, 2023.

Y. Liu, L. Kong, J. Cen, R. Chen, W. Zhang, L. Pan, K. Chen, and Z. Liu, “Segment any point cloud sequences by distilling vision foundation models,” Advances in Neural Information Processing Systems, vol. 36, 2024.

Y. Xie, Z. Huang, S. Shen, and J. Ma, “Semi-sd: Semi-supervised metric depth estimation via surrounding cameras for autonomous driving,” arXiv preprint arXiv:2503.19713, 2025.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213–3223.

F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2636–2645.