Generalization of Real-Time Motion Control with DRL Using Conditional Rewards and Symmetry Constraints
Resumo
Deep Reinforcement Learning has been increasingly explored as a method for generating physics-based motions in articulated characters. However, effective control tools are still necessary to better guide the learning process and provide animators with greater control and reliability over the resulting animations. This paper proposes new control tools, including the generalization of real-time control, conditional rewards, symmetry constraints, and a user interface. Real-time control allows dynamic adjustment of chosen parameters, conditional rewards simplify the competition between rewards, symmetry constraints reduce uncoordinated movements, and the user interface facilitates training and animation parameter specification. The proposed control tools show promise in improving the quality and control of physics-based character animation.
Palavras-chave:
Physics-Based Character Animation, Motion Control, Deep Reinforcement Learning, Real-Time Control
Referências
Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. 2019. DReCon: data-driven responsive control of physics-based characters. ACM Trans. Graph. 38, 6, Article 206 (2019), 11 pages. DOI: 10.1145/3355089.3356536
Nuttapong Chentanez, Matthias Müller, Miles Macklin, Viktor Makoviychuk, and Stefan Jeschke. 2018. Physics-based motion capture imitation with deep reinforcement learning. In Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG ’18). ACM, Article 1, 10 pages.
Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2009. Robust task-based control policies for physics-based characters. In ACM SIGGRAPH Asia 2009 papers. ACM, Yokohama, Japan, 1–9.
Antonio Santos de Sousa, Rubens Fernandes Nunes, Creto Augusto Vidal, Joaquim Bento Cavalcante-Neto, and Danilo Borges da Silva. 2021. Physics-Based Motion Control Through DRL’s Reward Functions. In Symposium on Virtual and Augmented Reality. IEEE, 127–136.
Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. 2022. Adversarial motion priors make good substitutes for complex reward functions. IEEE. In International Conference on Intelligent Robots and Systems (IROS), Vol. 2.
Vihanga Gamage, Cathy Ennis, and Robert Ross. 2021. Data-Driven Reinforcement Learning for Virtual Character Animation Control. arXiv:2104.06358 [cs.LG]
Thomas Geijtenbeek and Nicolas Pronost. 2012. Interactive character animation using simulated physics: A state-of-the-art review. In Computer graphics forum, Vol. 31. Wiley Online Library, 2492–2515.
Thomas Geijtenbeek, Michiel van de Panne, and A. Frank van der Stappen. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics (TOG) 32, 6, Article 206 (2013), 11 pages.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). PMLR, 1861–1870.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. 2017. Emergence of Locomotion Behaviours in Rich Environments. arXiv:1707.02286
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. 2020. Unity: A General Platform for Intelligent Agents. arXiv:1809.02627 [cs.LG]
Ariel Kwiatkowski, Eduardo Alvarado, Vicky Kalogeiton, C Karen Liu, Julien Pettré, Michiel van de Panne, and Marie-Paule Cani. 2022. A survey on reinforcement learning methods in character animation. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 613–639.
Sicen Li, Yiming Pang, Panju Bai, Zhaojin Liu, Jiawei Li, Shihao Hu, Liquan Wang, and Gang Wang. 2023. Learning Agility and Adaptive Legged Locomotion via Curricular Hindsight Reinforcement Learning. arXiv:2310.15583
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous Control with Deep Reinforcement Learning. arXiv (2015). arXiv:1509.02971
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37, 4 (2018), 1–14.
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018. SFV: Reinforcement Learning of Physical Skills from Videos. ACM Trans. Graph. 37, 6, Article 178 (Nov. 2018), 14 pages.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021. AMP: adversarial motion priors for stylized physics-based character control. ACM Trans. Graph. 40, 4, Article 144 (2021), 20 pages. DOI: 10.1145/3450626.3459670
Jiawei Ren, Cunjun Yu, Siwei Chen, Xiao Ma, Liang Pan, and Ziwei Liu. 2023. DiffMimic: Efficient Motion Mimicking with Differentiable Physics. arXiv:2304.03274 [cs.CV]
Yujin Tang, Jie Tan, and Tatsuya Harada. 2020. Learning Agile Locomotion via Adversarial Training. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 6098–6105.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2022. Physics-based character controllers using conditional VAEs. ACM Trans. Graph. 41, 4, Article 96 (2022), 12 pages. DOI: 10.1145/3528223.3530067
Pawel Wrotek, Odest Chadwicke Jenkins, and Morgan McGuire. 2006. Dynamo: dynamic, data-driven character control with adjustable balance. In Proceedings of the 2006 ACM SIGGRAPH symposium on Videogames. ACM, 61–70.
Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, and Florian Shkurti. 2021. Physics-based human motion estimation and synthesis from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11532–11541.
Zeshi Yang and Zhiqi Yin. 2021. Efficient Hyperparameter Optimization for Physics-based Character Animation. Proc. ACM Comput. Graph. Interact. Tech. 4, 1, Article 11 (apr 2021), 19 pages. DOI: 10.1145/3451254
Zhiqi Yin, Zeshi Yang, Michiel Van De Panne, and Kangkang Yin. 2021. Discovering diverse athletic jumping strategies. ACM Trans. Graph. 40, 4, Article 91 (2021), 17 pages. DOI: 10.1145/3450626.3459817
Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-Energy Locomotion. ACM Trans. Graph. 37, 4, Article 144 (jul 2018), 12 pages. DOI: 10.1145/3197517.3201397
Victor B Zordan and Jessica K Hodgins. 2002. Motion capture-driven simulations that hit and react. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer animation. 89–96.
Nuttapong Chentanez, Matthias Müller, Miles Macklin, Viktor Makoviychuk, and Stefan Jeschke. 2018. Physics-based motion capture imitation with deep reinforcement learning. In Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG ’18). ACM, Article 1, 10 pages.
Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2009. Robust task-based control policies for physics-based characters. In ACM SIGGRAPH Asia 2009 papers. ACM, Yokohama, Japan, 1–9.
Antonio Santos de Sousa, Rubens Fernandes Nunes, Creto Augusto Vidal, Joaquim Bento Cavalcante-Neto, and Danilo Borges da Silva. 2021. Physics-Based Motion Control Through DRL’s Reward Functions. In Symposium on Virtual and Augmented Reality. IEEE, 127–136.
Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. 2022. Adversarial motion priors make good substitutes for complex reward functions. IEEE. In International Conference on Intelligent Robots and Systems (IROS), Vol. 2.
Vihanga Gamage, Cathy Ennis, and Robert Ross. 2021. Data-Driven Reinforcement Learning for Virtual Character Animation Control. arXiv:2104.06358 [cs.LG]
Thomas Geijtenbeek and Nicolas Pronost. 2012. Interactive character animation using simulated physics: A state-of-the-art review. In Computer graphics forum, Vol. 31. Wiley Online Library, 2492–2515.
Thomas Geijtenbeek, Michiel van de Panne, and A. Frank van der Stappen. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics (TOG) 32, 6, Article 206 (2013), 11 pages.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). PMLR, 1861–1870.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. 2017. Emergence of Locomotion Behaviours in Rich Environments. arXiv:1707.02286
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. 2020. Unity: A General Platform for Intelligent Agents. arXiv:1809.02627 [cs.LG]
Ariel Kwiatkowski, Eduardo Alvarado, Vicky Kalogeiton, C Karen Liu, Julien Pettré, Michiel van de Panne, and Marie-Paule Cani. 2022. A survey on reinforcement learning methods in character animation. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 613–639.
Sicen Li, Yiming Pang, Panju Bai, Zhaojin Liu, Jiawei Li, Shihao Hu, Liquan Wang, and Gang Wang. 2023. Learning Agility and Adaptive Legged Locomotion via Curricular Hindsight Reinforcement Learning. arXiv:2310.15583
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous Control with Deep Reinforcement Learning. arXiv (2015). arXiv:1509.02971
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37, 4 (2018), 1–14.
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018. SFV: Reinforcement Learning of Physical Skills from Videos. ACM Trans. Graph. 37, 6, Article 178 (Nov. 2018), 14 pages.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021. AMP: adversarial motion priors for stylized physics-based character control. ACM Trans. Graph. 40, 4, Article 144 (2021), 20 pages. DOI: 10.1145/3450626.3459670
Jiawei Ren, Cunjun Yu, Siwei Chen, Xiao Ma, Liang Pan, and Ziwei Liu. 2023. DiffMimic: Efficient Motion Mimicking with Differentiable Physics. arXiv:2304.03274 [cs.CV]
Yujin Tang, Jie Tan, and Tatsuya Harada. 2020. Learning Agile Locomotion via Adversarial Training. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 6098–6105.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2022. Physics-based character controllers using conditional VAEs. ACM Trans. Graph. 41, 4, Article 96 (2022), 12 pages. DOI: 10.1145/3528223.3530067
Pawel Wrotek, Odest Chadwicke Jenkins, and Morgan McGuire. 2006. Dynamo: dynamic, data-driven character control with adjustable balance. In Proceedings of the 2006 ACM SIGGRAPH symposium on Videogames. ACM, 61–70.
Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, and Florian Shkurti. 2021. Physics-based human motion estimation and synthesis from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11532–11541.
Zeshi Yang and Zhiqi Yin. 2021. Efficient Hyperparameter Optimization for Physics-based Character Animation. Proc. ACM Comput. Graph. Interact. Tech. 4, 1, Article 11 (apr 2021), 19 pages. DOI: 10.1145/3451254
Zhiqi Yin, Zeshi Yang, Michiel Van De Panne, and Kangkang Yin. 2021. Discovering diverse athletic jumping strategies. ACM Trans. Graph. 40, 4, Article 91 (2021), 17 pages. DOI: 10.1145/3450626.3459817
Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-Energy Locomotion. ACM Trans. Graph. 37, 4, Article 144 (jul 2018), 12 pages. DOI: 10.1145/3197517.3201397
Victor B Zordan and Jessica K Hodgins. 2002. Motion capture-driven simulations that hit and react. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer animation. 89–96.
Publicado
30/09/2024
Como Citar
OLIVEIRA, Luis Ilderlandio da Silva; NUNES, Rubens Fernandes; VIDAL, Creto Augusto; CAVALCANTE-NETO, Joaquim Bento.
Generalization of Real-Time Motion Control with DRL Using Conditional Rewards and Symmetry Constraints. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 26. , 2024, Manaus/AM.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 103-112.
