Fooling the Model, Failing the Road: Benchmarking Inertial Sensor Fidelity in Driving Simulators
Abstract
CARLA and SUMO are widely adopted open-source simulators in autonomous driving and intelligent transportation systems. While neither was initially designed to generate high-fidelity inertial data such as that from an Inertial Measurement Unit (IMU), both have been employed in contexts where such data is critical. In this study, we evaluate the quality of inertial data produced by CARLA and SUMO through direct comparison with the naturalistic UAH-DriveSet dataset. Our analysis reveals that SUMO performs better in qualitative and quantitative assessments. To our knowledge, this is the first systematic comparison of its kind, highlighting limitations in current simulation tools and offering insights for autonomy-related studies using simulated inertial data.
References
Alvarez Lopez, P., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flötteröd, Y.-P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P., and Wießner, E. (2018). Microscopic Traffic Simulation using SUMO. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 2575–2582. IEEE.
Brahim, S. B., Ghazzai, H., Besbes, H., and Massoud, Y. (2022). A Machine Learning Smartphone-based Sensing for Driver Behavior Classification. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pages 610–614.
Chah, B., Lombard, A., Mualla, Y., Bkakria, A., Abbas-Turki, A., and Yaich, R. (2024). Building a Database of Simulated Driver Behaviors Using the SUMO Simulator. In Arai, K., editor, Intelligent Systems and Applications, pages 536–555, Cham. Springer Nature Switzerland.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017). CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16.
Gómez-Huélamo, C., Del Egido, J., Bergasa, L. M., Barea, R., López-Guillén, E., Arango, F., Araluce, J., and López, J. (2021). Train Here, Drive There: Simulating Real-World Use Cases with Fully-Autonomous Driving Architecture in CARLA Simulator. In Bergasa, L. M., Ocaña, M., Barea, R., López-Guillén, E., and Revenga, P., editors, Advances in Physical Agents II, pages 44–59, Cham. Springer International Publishing.
Gutiérrez-Moreno, R., Barea, R., López-Guillén, E., Araluce, J., and Bergasa, L. M. (2022). Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator. Sensors, 22(21).
Han, Y., Liu, Z., Sun, S., Li, D., Sun, J., Yuan, C., and Jr, M. H. A. (2024). CARLA-Loc: Synthetic SLAM Dataset with Full-stack Sensor Setup in Challenging Weather and Dynamic Environments.
Kullback, S. and Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22(1):79–86.
Lopez, P. A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flötteröd, Y.-P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P., and Wießner, E. (2018). Microscopic traffic simulation using sumo. In The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE.
Romera, E., Bergasa, L. M., and Arroyo, R. (2016). Need data for driver behaviour analysis? Presenting the public UAH-DriveSet. In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), page 387–392. IEEE Press.
Sarker, S., Maples, B., Islam, I., Fan, M., Papadopoulos, C., and Li, W. (2025). A Comprehensive Review on Traffic Datasets and Simulators for Autonomous Vehicles.
Schrader, M., Al Abdraboh, M., and Bittle, J. (2023). Comparing Measured Driver Behavior Distributions to Results from Car-Following Models using SUMO and Real-World Vehicle Trajectories from Radar: SUMO Default vs. Radar-Measured CF model Parameters. SUMO Conference Proceedings, 4:41–54.
Stević, S., Krunić, M., Dragojević, M., and Kaprocki, N. (2019). Development and Validation of ADAS Perception Application in ROS Environment Integrated with CARLA Simulator. In 2019 27th Telecommunications Forum (TELFOR), pages 1–4.
Brahim, S. B., Ghazzai, H., Besbes, H., and Massoud, Y. (2022). A Machine Learning Smartphone-based Sensing for Driver Behavior Classification. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pages 610–614.
Chah, B., Lombard, A., Mualla, Y., Bkakria, A., Abbas-Turki, A., and Yaich, R. (2024). Building a Database of Simulated Driver Behaviors Using the SUMO Simulator. In Arai, K., editor, Intelligent Systems and Applications, pages 536–555, Cham. Springer Nature Switzerland.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017). CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16.
Gómez-Huélamo, C., Del Egido, J., Bergasa, L. M., Barea, R., López-Guillén, E., Arango, F., Araluce, J., and López, J. (2021). Train Here, Drive There: Simulating Real-World Use Cases with Fully-Autonomous Driving Architecture in CARLA Simulator. In Bergasa, L. M., Ocaña, M., Barea, R., López-Guillén, E., and Revenga, P., editors, Advances in Physical Agents II, pages 44–59, Cham. Springer International Publishing.
Gutiérrez-Moreno, R., Barea, R., López-Guillén, E., Araluce, J., and Bergasa, L. M. (2022). Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator. Sensors, 22(21).
Han, Y., Liu, Z., Sun, S., Li, D., Sun, J., Yuan, C., and Jr, M. H. A. (2024). CARLA-Loc: Synthetic SLAM Dataset with Full-stack Sensor Setup in Challenging Weather and Dynamic Environments.
Kullback, S. and Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22(1):79–86.
Lopez, P. A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flötteröd, Y.-P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P., and Wießner, E. (2018). Microscopic traffic simulation using sumo. In The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE.
Romera, E., Bergasa, L. M., and Arroyo, R. (2016). Need data for driver behaviour analysis? Presenting the public UAH-DriveSet. In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), page 387–392. IEEE Press.
Sarker, S., Maples, B., Islam, I., Fan, M., Papadopoulos, C., and Li, W. (2025). A Comprehensive Review on Traffic Datasets and Simulators for Autonomous Vehicles.
Schrader, M., Al Abdraboh, M., and Bittle, J. (2023). Comparing Measured Driver Behavior Distributions to Results from Car-Following Models using SUMO and Real-World Vehicle Trajectories from Radar: SUMO Default vs. Radar-Measured CF model Parameters. SUMO Conference Proceedings, 4:41–54.
Stević, S., Krunić, M., Dragojević, M., and Kaprocki, N. (2019). Development and Validation of ADAS Perception Application in ROS Environment Integrated with CARLA Simulator. In 2019 27th Telecommunications Forum (TELFOR), pages 1–4.
Published
2025-09-29
How to Cite
FLORENCIO, Renan; FERNANDES, Silvio; COSTA, Paula D. Paro.
Fooling the Model, Failing the Road: Benchmarking Inertial Sensor Fidelity in Driving Simulators. In: NATIONAL MEETING ON ARTIFICIAL AND COMPUTATIONAL INTELLIGENCE (ENIAC), 22. , 2025, Fortaleza/CE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 1479-1490.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2025.12416.
