Drafting in Collectible Card Games via Reinforcement Learning
Resumo
Collectible card games (CCGs), such as Magic: the Gathering and Hearthstone, are played by tens of millions of players worldwide and are challenging for humans and artificial intelligence (AI) agents alike. To win, players must be proficient in two interdependent tasks: deck-building and battling. We present three deep reinforcement learning approaches for deck-building in the arena mode that differ in considering past information when making new choices. We formulate the problem in a game-agnostic manner and perform experiments on Legends of Code and Magic, a CCG designed for AI research. Results show that our trained draft agents outperform the currently best draft agents of the game and do so by building very different decks. Moreover, a Strategy Card Game AI competition participant improves from tenth to fourth place when using our best draft agent to build decks. This work is a step towards strong and fast game-playing AI for CCGs, one of the current academic AI milestones that would enable thorough playtesting of new cards before they are released – a long-standing problem in the CCG industry.
Referências
N. Brown and T. Sandholm, “Superhuman AI for multiplayer poker,” Science, vol. 365, no. 6456, pp. 885–890, 2019.
B. Hawley, “Checking in with play design,” 2019. [Online]. Available: [link].
J. Kowalski and R. Miernik, “Evolutionary approach to collectible arena deckbuilding using active card game genes,” in CEC 2020. IEEE, 2020.
J. Kowalski and R. Miernik, “Legends of Code and Magic,” https://legendsofcodeandmagic.com, 2018, accessed: 2020-03-27.
G. Brockman et al., “OpenAI Gym,” CoRR, vol. abs/1606.01540, 2016.
R. Vieira, A. R. Tavares, and L. Chaimowicz, “Drafting in collectible card games via reinforcement learning,” Master’s thesis, UFMG, 2020.
S. Pal and S. Mitra, “Multilayer perceptron, fuzzy sets, and classification.” IEEE transactions on neural networks, vol. 3, no. 5, 1992.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
CodinGame community, “Legends of Code & Magic (CC05) - Feedback & Strategies,” [link], 2018.
B. Baker, I. Kanitscheider, T. M. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch, “Emergent tool use from multi-agent autocurricula,” in ICLR 2020, 2020.
J. Schulman et al., “Proximal policy optimization algorithms,” CoRR, vol. abs/1707.06347, 2017.
A. Hill et al., “Stable baselines,” https://github.com/hill-a/stable-baselines, 2018.
R. Vieira, L. Chaimowicz, and A. R. Tavares, “OpenAI Gym environments for Legends of Code and Magic,” https://github.com/ronaldosvieira/gym-locm, 2019.
J. Bergstra, D. Yamins, and D. D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in ICML 2013. JMLR.org, 2013.
I. T. Jolliffe and J. Cadima, “Principal component analysis: a review and recent developments,” Phil. Trans. A, vol. 374, no. 2065, 2016.
S. P. Lloyd, “Least squares quantization in PCM,” IEEE Trans. Inf. Theory, vol. 28, no. 2, 1982.
P. J. Rousseeuw, “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis,” Journal of computational and applied mathematics, vol. 20, pp. 53–65, 1987.
R. Vieira, L. Chaimowicz, and A. R. Tavares, “Reinforcement learning in collectible card games: Preliminary results on Legends of Code and Magic,” in SBGames 2019, 2019.
R. Vieira, A. R. Tavares, and L. Chaimowicz, “Drafting in collectible card games via reinforcement learning,” in SBGames 2020. IEEE, 2020.
