Analysis of Virtual Environments for the Evaluation of LiDAR-Based Object Detection Models: A Study Using the CARLA Simulator
Resumo
Virtual environments are increasingly used for testing LiDAR-based object detection models, offering advantages in cost, safety, and reproducibility. This paper presents an evaluation of the CARLA simulator as a virtual testbed to assess cross-domain performance of a detection model trained on the real-world nuScenes dataset. Using matching sensor parameters and urban layouts, we generated synthetic LiDAR point clouds in CARLA and tested the model under various simulated scenarios, including changes in lighting and object density. Results show that the mean Average Precision (mAP) in synthetic data was 5.2 percentage points lower than in the real domain, with performance degradation concentrated in classes affected by occlusion and sparse geometry. Despite this, certain object categories (e.g., vehicles and static infrastructure) maintained comparable detection rates across domains. The originality of this work lies in the controlled analysis of transferability using real-to-sim evaluation under matched configurations. Our findings support the use of simulation for early-stage validation, particularly in tasks requiring high spatial accuracy and reproducibility. This study highlights the practical trade-offs and domain limitations of using virtual environments as a reliable complement to physical testing in LiDAR-based 3D perception pipelines.
Referências
Y. Zhou, S. Wang, C. Li and Q. Xu, "Synthetic-to-Real Transfer Learning for LiDAR Perception: A Review," arXiv:2503.09803, 2024.
Y. Li, H. Zhu, F. Chen and J. Zhang, "Multi-Sensor Fusion and Transfer Learning in 3D Object Detection," arXiv:2312.12772, 2023.
CARLA Simulator, "CARLA Open-source Simulator for Autonomous Driving Research," v0.9.13, 2025. [Online]. Available: [link]
A. El Sallab et al. "Unsupervised Neural Sensor Models for Synthetic LiDAR Data Augmentation," arXiv:1911.10575, 2019.
J. Montalvo, P. Carballeira and Á. García-Martín, "SynthmanticLiDAR: A Synthetic Dataset for Semantic Segmentation on LiDAR Imaging," arXiv:2501.19035, 2025.
D. Dworak, M. Stanek and P. Skrzypczyński, "Performance of LiDAR Object Detection Deep Learning Architectures Based on Artificially Generated Point Cloud Data from CARLA Simulator," in Proc. 24th Int. Conf. Methods and Models in Automation and Robotics (MMAR), pp. 1-6, 2019. DOI: 10.1109/MMAR.2019.8864706.
M. Patel and R. Jung, "Simulation-based Performance Evaluation of 3D Object Detection Methods with Deep Learning for a LiDAR Point Cloud Dataset in a SOTIF-related Use Case," arXiv:2503.03548, 2025.
S. Skoog, "LiDAR Perception in a Virtual Environment Using Deep Learning," M.Sc. thesis, KTH Royal Institute of Technology, 2024. [Online]. Available: [link]
J. Zhang et al. "NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields," arXiv:2304.14811, 2023.
Y. Tao et al. "SiLVR: Scalable LiDAR-Visual Reconstruction with Neural Radiance Fields for Robotic Inspection," arXiv:2403.06877, 2024.
M. Silva et al. "Integração de um framework de realidade virtual com um sistema de simulação de objetos 3D para aplicações de treinamento médico," in Anais do XXI Simpósio de Realidade Virtual e Aumentada (SVR), 2019.
J.-E. Deschaud et al. "Paris-CARLA-3D: A Real and Synthetic Outdoor Point Cloud Dataset for Challenging Tasks in 3D Mapping," Remote Sensing, vol. 13, no. 22, p. 4713, 2021. DOI: 10.3390/rs13224713.
H. Hu et al. "Investigating the Impact of Multi-LiDAR Placement on Object Detection for Autonomous Driving," arXiv:2105.00373, 2021.
R. T. C. Vieira, J. A. Macedo and B. Feijó, "Navigation of Autonomous Agents in 3D Environments Using Deep Reinforcement Learning and Simulation," Anais do XXI Simpósio de Realidade Virtual e Aumentada (SVR), pp. 71-80, 2019.
H. Kussaba, E. Nascimento and M. F. M. Campos, "Deep Learning and Synthetic Environments for Visual Inspection with Drones," Anais do XXI Simpósio de Realidade Virtual e Aumentada (SVR), pp. 91-100, 2019.
H. Caesar et al. "nuScenes: A Multimodal Dataset for Autonomous Driving," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 11621-11631, 2020.
OpenPCDet Development Team, "OpenPCDet: An Open-Source Toolbox for 3D Object Detection from Point Clouds," 2023. [Online]. Available: [link]
Y. Yan et al. "SECOND: Sparsely Embedded Convolutional Detection," Sensors, vol. 18, no. 10, p. 3337, 2018. DOI: 10.3390/s18103337.
A. Dosovitskiy et al. "CARLA: An Open Urban Driving Simulator," in Proc. 1st Annual Conf. Robot Learning (CoRL), pp. 1-16, 2017.
G. B. Ribeiro, S. R. Gabriel and L. F. O. Martins, "Simulação de Ambientes Urbanos para Avaliação de Modelos de Navegação Autônoma," Anais do XXI Simpósio de Realidade Virtual e Aumentada (SVR), 2019.
