Methods and Algorithms for Knowledge Reuse in Multiagent Reinforcement Learning

• Felipe Leno da Silva USP
• Anna Helena Reali Costa USP

DOI: https://doi.org/10.5753/ctd.2020.11360

Resumo

Reinforcement Learning (RL) is a powerful tool that has been used to solve increasingly complex tasks. RL operates through repeated interactions of the learning agent with the environment, via trial and error. However, this learning process is extremely slow, requiring many interactions. In this thesis, we leverage previous knowledge so as to accelerate learning in multiagent RL problems. We propose knowledge reuse both from previous tasks and from other agents. Several flexible methods are introduced so that each of these two types of knowledge reuse is possible. This thesis adds important steps towards more flexible and broadly applicable multiagent transfer learning methods.

Palavras-chave: Multiagent Reinforcement Learning, Transfer Learning, Reinforcement Learning, Multiagent Systems

Referências

Bellemare, M. G., Dabney, W., and Munos, R. (2017). A distributional perspective on reinforcement learning. CoRR, abs/1707.06887.

Omidshafiei, S., Kim, D., Liu, M., Tesauro, G., Riemer, M., Amato, C., Campbell, M., and How, J. P. (2019). Learning to Teach in Cooperative Multiagent Reinforcement Learning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence.

Silva, F. L. D. and Costa, A. H. R. (2017). Towards Zero-Shot Autonomous Inter-Task Mapping through Object-Oriented Task Description. In Workshop on Transfer in Reinforcement Learning (TiRL).

Silva, F. L. D. and Costa, A. H. R. (2018). Object-Oriented Curriculum Generation for Reinforcement Learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1026-1034.

Silva, F. L. D. and Costa, A. H. R. (2019). A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems. Journal of Artificial Intelligence Research (JAIR), 69:645-703.

Silva, F. L. D., Costa, A. H. R., and Stone, P. (2019a). Building Self-Play Curricula Online by Playing with Expert Agents. In Proceedings of the 8th Brazilian Conference on Intelligent Systems (BRACIS).

Silva, F. L. D. et al. (2020a). Agents Teaching Agents: A Survey on Inter-agent Transfer Learning. Autonomous Agents and Multi-Agent Systems, 34(1):9.

Silva, F. L. D., Glatt, R., and Costa, A. H. R. (2016). Object-Oriented Reinforcement Learning in Cooperative Multiagent Domains. In Proceedings of the 5th Brazilian Conference on Intelligent Systems (BRACIS), pages 19-24.

Silva, F. L. D., Glatt, R., and Costa, A. H. R. (2017). Simultaneously Learning and Advising in Multiagent Reinforcement Learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1100-1108.

Silva, F. L. D., Glatt, R., and Costa, A. H. R. (2019b). MOO-MDP: An Object- Oriented Representation for Cooperative Multiagent Reinforcement Learning. IEEE Transactions on Cybernetics, 49(2):567-579.

Silva, F. L. D., Hernandez-Leal, P., Kartal, B., and Taylor, M. E. (2020b). Uncertainty-Aware Action Advising for Deep Reinforcement Learning Agents. In Proceedings of the 34th AAAI Conference on Artificial Intelligence.

Silva, F. L. D., Nishida, C. E. H., Roijers, D. M., and Costa, A. H. R. (2020c). Coordination of Electric Vehicle Charging through Multiagent Reinforcement Learning. IEEE Transactions on Smart Grid, (accepted).

Silva, F. L. D., Taylor, M. E., and Costa, A. H. R. (2018). Autonomously Reusing Knowledge in Multiagent Reinforcement Learning. In International Joint Conference on Artificial Intelligence (IJCAI), pages 5487-5493.

Stone, P., Kaminka, G. A., Kraus, S., and Rosenschein, J. S. (2010). Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination. In AAAI Conference on Artificial Intelligence (AAAI), pages 1504-1509.

Tan, M. (1993). Multi-agent Reinforcement Learning: Independent vs. Cooperative Agents. In International Conference on Machine Learning (ICML), pages 330-337.

Taylor, A., Dusparic, I., Galvan-Lopez, E., Clarke, S., and Cahill, V. (2014). Accelerating Learning in Multi-Objective Systems through Transfer Learning. In International Joint Conference on Neural Networks, pages 2298-2305.

Taylor, M. E. and Stone, P. (2009). Transfer Learning for Reinforcement Learning Domains: A Survey. Journal of Machine Learning Research (JMLR), 10:1633-1685.

Torrey, L. and Taylor, M. E. (2013). Teaching on a Budget: Agents Advising Agents in Reinforcement Learning. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1053-1060.