Face-capture with automatic blendshape generation
Resumo
The goal of this paper is the development of a system for the production of performance-driven facial animation that automatically generates a blendshape model from input video frames. This simplifies the production of this kind of model to be used in animation projects. The proposed method is designed to be used without the need for expensive hardware, such that any computer with a webcam can run the system.
Referências
L. Williams, "Performance-driven facial animation," in Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, ser. SIGGRAPH '90. New York, NY, USA: Association for Computing Machinery, 1990, p. 235-242. [Online]. Available: https://doi.org/10.1145/97879.97906
Z. Deng and J. Noh, Computer Facial Animation: A Survey. London: Springer London, 2008, pp. 1-28. [Online]. Available: https://doi.org/10.1007/978-1-84628-907-1_1
V. Orvalho, P. Bastos, F. Parke, B. Oliveira, and X. Alvarez, "A Facial Rigging Survey," in Eurographics 2012 - State of the Art Reports, M.-P. Cani and F. Ganovelli, Eds. The Eurographics Association, 2012.
F. I. Parke and K. Waters, Computer Facial Animation. A K Peters, 2008.
J. P. Lewis, K. Anjyo, T. Rhee, M. Zhang, F. Pighin, and Z. Deng, "Practice and Theory of Blendshape Facial Models," in Eurographics 2014 - State of the Art Reports, S. Lefebvre and M. Spagnuolo, Eds. The Eurographics Association, 2014.
C. Cao, Y. Weng, S. Lin, and K. Zhou, "3d shape regression for real-time facial animation," ACM Trans. Graph., vol. 32, no. 4, pp. 41:1-41:10, Jul. 2013. [Online]. Available: http://doi.acm.org/10.1145/2461912.2462012
C. Cao, D. Bradley, K. Zhou, and T. Beeler, "Real-time high-fidelity facial performance capture," ACM Trans. Graph., vol. 34, no. 4, pp. 46:1-46:9, Jul. 2015. [Online]. Available: http://doi.acm.org/10.1145/2766943
Y. Weng, C. Cao, Q. Hou, and K. Zhou, "Real-time facial animation on mobile devices," Graphical Models, vol. 76, no. 3, pp. 172 - 179, 2014, computational Visual Media Conference 2013. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1524070313000295
J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner, "Demo of face2face: Real-time face capture and reenactment of rgb videos," in ACM SIGGRAPH 2016 Emerging Technologies, ser. SIGGRAPH '16. New York, NY, USA: ACM, 2016, pp. 5:1-5:2. [Online]. Available: http://doi.acm.org/10.1145/2929464.2929475
T. Weise, S. Bouaziz, H. Li, and M. Pauly, "Realtime performance-based facial animation," ACM Trans. Graph., vol. 30, no. 4, pp. 77:1-77:10, Jul. 2011. [Online]. Available: http://doi.acm.org/10.1145/2010324.1964972
Y.-L. Chen, H.-T. Wu, F. Shi, X. Tong, and J. Chai, "Accurate and robust 3d facial capture using a single rgbd camera." in ICCV. IEEE Computer Society, 2013, pp. 3615-3622. [Online]. Available: [link]
H. Li, J. Yu, Y. Ye, and C. Bregler, "Realtime facial animation with on-the-fly correctives," ACM Trans. Graph., vol. 32, no. 4, pp. 42:1-42:10, Jul. 2013. [Online]. Available: http://doi.acm.org/10.1145/2461912.2462019
S. Bouaziz, Y. Wang, and M. Pauly, "Online modeling for realtime facial animation," ACM Trans. Graph., vol. 32, no. 4, pp. 40:1-40:10, Jul. 2013. [Online]. Available: http://doi.acm.org/10.1145/2461912.2461976
P.-L. Hsieh, C. Ma, J. Yu, and H. Li, "Unconstrained realtime facial performance capture." in CVPR. IEEE Computer Society, 2015, pp. 1675-1683. [Online]. Available: http://dblp.uni-trier.de/db/conf/cvpr/cvpr2015.html#HsiehMYL15
B. Choe, H. Lee, and H. Ko, "Performance-driven muscle-based facial animation," Comput. Animat. Virtual Worlds, vol. 12, pp. 67-79, 2001.
Z. Deng, P.-Y. Chiang, P. Fox, and U. Neumann, "Animating blendshape faces by cross-mapping motion capture data," in Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games, ser. I3D '06. New York, NY, USA: Association for Computing Machinery, 2006, p. 43-48. [Online]. Available: https://doi.org/10.1145/1111411.1111419
F. Pighin, R. Szeliski, and D. Salesin, "Resynthesizing facial animation through 3d model-based tracking," in Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, 1999, pp. 143-150 vol.1.
B. Choe and H.-S. Ko, "Analysis and synthesis of facial expressions with hand-generated muscle actuation basis," in Proceedings Computer Animation 2001. Fourteenth Conference on Computer Animation (Cat. No.01TH8596), 2001, pp. 12-19.
T. Rhee, Y. Hwang, J. D. Kim, and C. Kim, "Real-time facial animation from live video tracking," in Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ser. SCA '11. New York, NY, USA: Association for Computing Machinery, 2011, p. 215-224. [Online]. Available: https://doi.org/10.1145/2019406.2019435
V. Blanz, C. Basso, T. Poggio, and T. Vetter, "Reanimating faces in images and video," Comput. Graph. Forum, vol. 22, pp. 641-650, 09 2003.
E. Chuang, F. Deshpande, and C. Bregler, "Facial expression space learning," in 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings., 2002, pp. 68-76.
Anzovin. (2012) Face machine nodes. [Online]. Available: https://anzovin.squarespace.com/tfmn
Animationsinstitut. (2012) Facial animation toolset. [Online]. Available: https://animationsinstitut.de/en/research/tools/facial-animation-toolset
Scriptattack. (2012) Bony face: Facial animation system for 3dsmax. [Online]. Available: http://www.scriptattack.com/maxscripts/bonyface/index_eng.html
Google. (2020) Mediapipe. [Online]. Available: https://mediapipe.dev
Z. Deng and J. Noh, Computer Facial Animation: A Survey. London: Springer London, 2008, pp. 1-28. [Online]. Available: https://doi.org/10.1007/978-1-84628-907-1_1
V. Orvalho, P. Bastos, F. Parke, B. Oliveira, and X. Alvarez, "A Facial Rigging Survey," in Eurographics 2012 - State of the Art Reports, M.-P. Cani and F. Ganovelli, Eds. The Eurographics Association, 2012.
F. I. Parke and K. Waters, Computer Facial Animation. A K Peters, 2008.
J. P. Lewis, K. Anjyo, T. Rhee, M. Zhang, F. Pighin, and Z. Deng, "Practice and Theory of Blendshape Facial Models," in Eurographics 2014 - State of the Art Reports, S. Lefebvre and M. Spagnuolo, Eds. The Eurographics Association, 2014.
C. Cao, Y. Weng, S. Lin, and K. Zhou, "3d shape regression for real-time facial animation," ACM Trans. Graph., vol. 32, no. 4, pp. 41:1-41:10, Jul. 2013. [Online]. Available: http://doi.acm.org/10.1145/2461912.2462012
C. Cao, D. Bradley, K. Zhou, and T. Beeler, "Real-time high-fidelity facial performance capture," ACM Trans. Graph., vol. 34, no. 4, pp. 46:1-46:9, Jul. 2015. [Online]. Available: http://doi.acm.org/10.1145/2766943
Y. Weng, C. Cao, Q. Hou, and K. Zhou, "Real-time facial animation on mobile devices," Graphical Models, vol. 76, no. 3, pp. 172 - 179, 2014, computational Visual Media Conference 2013. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1524070313000295
J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner, "Demo of face2face: Real-time face capture and reenactment of rgb videos," in ACM SIGGRAPH 2016 Emerging Technologies, ser. SIGGRAPH '16. New York, NY, USA: ACM, 2016, pp. 5:1-5:2. [Online]. Available: http://doi.acm.org/10.1145/2929464.2929475
T. Weise, S. Bouaziz, H. Li, and M. Pauly, "Realtime performance-based facial animation," ACM Trans. Graph., vol. 30, no. 4, pp. 77:1-77:10, Jul. 2011. [Online]. Available: http://doi.acm.org/10.1145/2010324.1964972
Y.-L. Chen, H.-T. Wu, F. Shi, X. Tong, and J. Chai, "Accurate and robust 3d facial capture using a single rgbd camera." in ICCV. IEEE Computer Society, 2013, pp. 3615-3622. [Online]. Available: [link]
H. Li, J. Yu, Y. Ye, and C. Bregler, "Realtime facial animation with on-the-fly correctives," ACM Trans. Graph., vol. 32, no. 4, pp. 42:1-42:10, Jul. 2013. [Online]. Available: http://doi.acm.org/10.1145/2461912.2462019
S. Bouaziz, Y. Wang, and M. Pauly, "Online modeling for realtime facial animation," ACM Trans. Graph., vol. 32, no. 4, pp. 40:1-40:10, Jul. 2013. [Online]. Available: http://doi.acm.org/10.1145/2461912.2461976
P.-L. Hsieh, C. Ma, J. Yu, and H. Li, "Unconstrained realtime facial performance capture." in CVPR. IEEE Computer Society, 2015, pp. 1675-1683. [Online]. Available: http://dblp.uni-trier.de/db/conf/cvpr/cvpr2015.html#HsiehMYL15
B. Choe, H. Lee, and H. Ko, "Performance-driven muscle-based facial animation," Comput. Animat. Virtual Worlds, vol. 12, pp. 67-79, 2001.
Z. Deng, P.-Y. Chiang, P. Fox, and U. Neumann, "Animating blendshape faces by cross-mapping motion capture data," in Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games, ser. I3D '06. New York, NY, USA: Association for Computing Machinery, 2006, p. 43-48. [Online]. Available: https://doi.org/10.1145/1111411.1111419
F. Pighin, R. Szeliski, and D. Salesin, "Resynthesizing facial animation through 3d model-based tracking," in Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, 1999, pp. 143-150 vol.1.
B. Choe and H.-S. Ko, "Analysis and synthesis of facial expressions with hand-generated muscle actuation basis," in Proceedings Computer Animation 2001. Fourteenth Conference on Computer Animation (Cat. No.01TH8596), 2001, pp. 12-19.
T. Rhee, Y. Hwang, J. D. Kim, and C. Kim, "Real-time facial animation from live video tracking," in Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, ser. SCA '11. New York, NY, USA: Association for Computing Machinery, 2011, p. 215-224. [Online]. Available: https://doi.org/10.1145/2019406.2019435
V. Blanz, C. Basso, T. Poggio, and T. Vetter, "Reanimating faces in images and video," Comput. Graph. Forum, vol. 22, pp. 641-650, 09 2003.
E. Chuang, F. Deshpande, and C. Bregler, "Facial expression space learning," in 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings., 2002, pp. 68-76.
Anzovin. (2012) Face machine nodes. [Online]. Available: https://anzovin.squarespace.com/tfmn
Animationsinstitut. (2012) Facial animation toolset. [Online]. Available: https://animationsinstitut.de/en/research/tools/facial-animation-toolset
Scriptattack. (2012) Bony face: Facial animation system for 3dsmax. [Online]. Available: http://www.scriptattack.com/maxscripts/bonyface/index_eng.html
Google. (2020) Mediapipe. [Online]. Available: https://mediapipe.dev
Publicado
24/10/2022
Como Citar
NOGUEIRA, João Vitor; SUMAR, Bruno; CARVALHO, Leonardo.
Face-capture with automatic blendshape generation. In: WORKSHOP DE TRABALHOS DA GRADUAÇÃO - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 35. , 2022, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 96-100.
DOI: https://doi.org/10.5753/sibgrapi.est.2022.23269.
