Does a Q-Learning NetLogo Extension Simplify the Development of Agent-based Simulations?

• Eloísa Bazzanella UDESC
• Fernando Santos UDESC

DOI: https://doi.org/10.5753/wesaac.2021.33403

Resumo

Agent-based modeling and simulation is a simulation paradigm that allows focusing on individuals, their interactions, and the complex behavior that emerges from them. Agent-based simulations are typically developed in simulation platforms that provide features related to agents. One such platform is NetLogo, to which a reinforcement learning extension was made available recently. The extension provides commands for using the Q-Learning algorithm, but no evaluation on whether it simplifies the development of simulations is available. This paper presents a quantitative evaluation on using the extension in two simulations: the classic cliff walking problem; and a real-world, adaptive traffic signal control (ATSC) simulation. Results show that the size of simulations source code developed using the extension is smaller than those developed without using it, giving evidence that the extension simplifies the development of simulations

Referências

Allan J Albrecht. Measuring application development productivity. In Proceedings of the joint SHARE/GUIDE/IBM application development symposium, volume 10, pages 83–92, 1979.

Ana L. C. Bazzan. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multiagent Systems, 18(3): 342–375, June 2009.

Ana L. C. Bazzan and Franziska Klügl. A review on agent-based technology for traffic and transportation. The Knowledge Engineering Review, FirstView:1–29, 4 2013.

Barry Boehm, Bradford Clark, Ellis Horowitz, Chris Westland, Ray Madachy, and Richard Selby. Cost models for future software life cycle processes: COCOMO 2.0. Annals of Software Engineering, 1(1):57–94, 1995.

CoMSES. CoMSES Catalog, 2020. URL [link]. Acesso em: Jul/2020.

Franziska Klügl and Ana L. C. Bazzan. Agent-based modeling and simulation. AI Magazine, 33(3):29–40, 2012.

Kevin Kons. Biblioteca Q-Learning para desenvolvimento de simulações com agentes na plataforma NetLogo. Trabalho de conclusão de curso, Universidade do Estado de Santa Catarina (UDESC), 2019.

Kevin Kons and Fernando Santos. Cliff walking with q-learning netlogo extension. CoMSES Computational Model Library, 2019. Retrieved from: [link].

Charles Macal and Michael North. Introductory tutorial: Agent-based modeling and simulation. In Proceedings of the 2014 Winter Simulation Conference, WSC ’14, pages 6–20, Piscataway, NJ, USA, 2014. IEEE Press.

Patrick Mannion, Jim Duggan, and Enda Howley. An experimental review of reinforcement learning algorithms for adaptive traffic signal control. In Leo Thomas McCluskey, Apostolos Kotsialos, P. Jörg Müller, Franziska Klügl, Omer Rana, and René Schumann, editors, Autonomic Road Transport Support Systems, pages 47–66. Springer, 2016.

S. T. Monteiro and C. H. C. Ribeiro. Desempenho de algoritmos de aprendizagem por reforço sob condições de ambiguidade sensorial em robótica móvel. Revista Controle & Automação, 15(3):320–338, 2004.

D. de. Oliveira and A. L. C. Bazzan. Multiagent learning on traffic lights control: effects of using shared information. IGI Global, pages 307–321, 2009.

Stuart Russel and Peter Norvig. Inteligência Artificial. Rio de Janeiro: Campus, 2 edition, 2004.

Fernando Santos, Ingrid Nunes, and Ana L. C. Bazzan. Model-driven agent-based simulation development: a modeling language and empirical evaluation in the adaptive traffic signal control domain. Simulation Modelling Practice and Theory, 83: 162–187, April 2018.

Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2 edition, 2018.

Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine learning, 33 (3–4):279–292, 1992.

Uri Wilensky. NetLogo, 1999. URL [link]. Center for Connected Learning and Computer-Based Modeling, Northwestern University. Evanston, IL.

Michael Wooldridge. An introduction to multiagent systems. John Wiley & Sons, 2009.