Closed-Loop Evaluation of Bird’s-Eye-View Maps from Cross-View Transformers as Inputs to Behavior-Cloning Policies
Resumo
In autonomous driving, Bird’s-Eye View (BEV) representations provide a structured, top-down abstraction of the vehicle’s surroundings and have become a key input modality for Behavioral Cloning (BC) policies. While ground-truth BEV maps are readily available in simulation, real-world deployment requires replacing them with camera-predicted counterparts—a substitution that introduces perceptual errors whose downstream impact on closed-loop driving performance is not well understood. In this work, we investigate the use of Cross-View Transformer (CVT)-predicted BEV maps as direct policy inputs for a BC agent in the CARLA simulator. We propose a six-channel BEV representation covering road surface, planned route, lane boundaries, vehicles, pedestrians, and traffic lights, and introduce a Kernel Density Estimation (KDE) weighting scheme that rebalances the segmentation loss towards underrepresented driving maneuvers such as curves and intersections. Closed-loop evaluation across two CARLA towns shows that the KDE-weighted model is the only predicted-BEV agent to complete a full episode without infractions, despite not achieving the highest aggregate IoU. This discrepancy reveals that global segmentation metrics are poor proxies for driving performance: what determines navigation success is prediction quality at geometrically critical locations, and the route channel emerges as the primary bottleneck for reliable agent navigation under predicted BEV inputs.
Referências
Antonelo, E. A., Couto, G. C. K., Möller, C., and Fernandes, P. H. (2024). Investigating behavior cloning from few demonstrations for autonomous driving based on bird’s-eye view in simulated cities. In BRACIS, pages 155–168. Springer.
Bansal, M., Krizhevsky, A., and Ogale, A. (2018). Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst.
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., and Zieba, K. (2016). End to end learning for self-driving cars.
Codevilla, F., Müller, M., Dosovitskiy, A., López, A. M., and Koltun, V. (2017). End-to-end driving via conditional imitation learning. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–9.
Couto, G. C. K. and Antonelo, E. A. (2023). Hierarchical generative adversarial imitation learning with mid-level input generation for autonomous driving on urban environments. arXiv preprint arXiv:2302.04823.
Ota, K., Jha, D. K., and Kanezaki, A. (2021). Training larger networks for deep reinforcement learning. arXiv preprint arXiv:2102.07920.
Park, S., Hong, S., Kim, S., and Lee, K. (2023). Cross-view transformers for real-time map-view semantic segmentation. In IEEE/CVF CVPR.
Petrazzini, I. G. and Antonelo, E. A. (2021). Proximal policy optimization with continuous bounded action space via the beta distribution. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1–8. IEEE.
Philion, J. and Fidler, S. (2020). Lift, splat, shoot: A multicamera bird’s-eye view for 3d object detection. In European Conference on Computer Vision (ECCV).
Pomerleau, D. A. (1988). Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1.
Santos, F. C. d., Antonelo, E. A., and Couto, G. C. K. (2025). An initial study of bird’s-eye view generation for autonomous vehicles using cross-view transformers. arXiv preprint arXiv:2508.12520.
Tan, M. and Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105–6114. PMLR.
