From Static Plans to Self-Evolving Policies: Migrating Legacy Traffic-Control Software to Adaptive Reinforcement Learning

• Jeferson Silva IFBA
• Isaque Andrade IFBA
• Cleia Libarino IFBA
• João Erivando Soares Marques IFBA
• Luis A. Correia Filho IFBA
• Kenedy M. Geraldo dos Santos IFBA
• José Alberto Diaz Amado IFBA
• Crescencio Lima IFBA

DOI: https://doi.org/10.5753/vem.2026.32473

Resumo

Traffic signal control is, in practice, a software maintenance problem: each signalized intersection is governed by a static, manually maintained control artifact—a fixed-time signal plan hard-coded into the controller’s configuration. We frame the modernization of such systems as a problem of software migration and renovation, replacing the static artifact with a self-evolving control policy learned by a Reinforcement Learning (RL) agent, while preserving the system’s external interfaces. The legacy artifact is reconstructed rigorously from the Brazilian DENATRAN standard via Webster’s method, so the migration is judged against current engineering practice rather than a strawman. Two renovated variants are evaluated—tabular Q-Learning and a Deep Q-Network (DQN)— in scenarios of increasing complexity (an isolated intersection and a coordinated arterial corridor) using the SUMO simulator. Because a learned policy exposes no inspectable source, we treat the visualization of its convergence and runtime behavior as a first-class instrument for comprehending and validating the evolved software. The renovated DQN artifact reduced waiting time by 58.8% and stops by 24.8% at the isolated intersection, and reduced waiting time by 18.9% on the corridor, while behavioral visualizations exposed a fairness/throughput trade-off that aggregate metrics alone would have misread. The results support the migration of static control logic to maintainable, self-evolving artifacts.

Palavras-chave: Software migration, Software renovation, traffic-control, reinforcement learning

Referências

Michael Behrisch et al. 2011. SUMO–Simulation of UrbanMObility: AnOverview. In Proceedings of SIMUL 2011.

Tse-Leung Chen et al. 2022. Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2201.10803 (2022).

Youqing Chen et al. 2023. Traffic signal optimization control method based on adaptive weighted averaged double deep q network. Applied Intelligence (2023), 1–22.

DENATRAN. 2020. Manual Brasileiro de Sinalização de Trânsito - Volume V: Sinalização Semafórica. Departamento Nacional de Trânsito.

Instituto de Pesquisa Econômica Aplicada. 2020. Mobilidade Urbana no Brasil. IPEA.

R. Kartikasari, G. Prakarsa, and D. Pradeka. 2020. Optimization of traffic light control using fuzzy logic sugeno method. International Journal of Global Operations Research 1 (2020).

J. Li et al. 2023. Distributed signal control of arterial corridors using multiagent deep reinforcement learning. Transportation Research Part C: Emerging Technologies (2023), 146–164.

Volodymyr Mnih et al. 2015. Human-level control through deep reinforcement learning. Nature 518 (2015), 529–533.

M. Muresan, L. Fu, and G. Pan. 2019. Adaptive traffic signal control with deep reinforcement learning: an exploratory investigation. arXiv preprint arXiv:1901.00960 (2019).

Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction (2nd ed.). MIT Press.

Michel Tokic. 2010. Adaptive 𝜖-greedy Exploration in Reinforcement Learning Based on Value Differences. In KI 2010: Advances in Artificial Intelligence.

Christopher J. C. H. Watkins and Peter Dayan. 1992. Q-learning. Machine Learning 8 (1992), 279–292.

Z. Zhu et al. 2023. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).