A Movement Analysis Application using Human Pose Estimation and Action Correction
Resumo
Human pose estimation (HPE) is an important field of computer vision that aims to predict poses of individuals from videos and images. It has been used in many different areas including human-computer interaction, motion analysis, surveillance, action prediction, action correction, augmented reality, virtual reality, and healthcare. Executing movements correctly is crucial in all kinds of physical activities, both to increase performance and reduce risk of injury. HPE is poised to help athletes better analyse the quality of their movements. This work proposes a model for movement analysis, repetition count, and movement correction in physical exercises using HPE. For this purpose, a study is carried out in the field of HPE applied to sports and another study is focused on HPE for correction and postural analysis. From this, it is verified what is the state of the art in HPE for physical exercises and what is the best method for analyzing movements. This work implements an application with improvements in respect to other related work, focusing mainly on the feedback presented to the user when performing a certain movement. To validate the proposed model, a quantitative research was carried out using the Unified Theory of Acceptance and Use (UTAUT). For both people who exercise and professionals in the field of physical education, the results demonstrate that the application is able to analyze the biomechanics of movement, responding with speed and precision to execution errors. Among other results are: user satisfaction, interest in using the application in the future, and agreement in relation to good performance in helping and analyzing physical exercises.
Palavras-chave:
human pose estimation, training assistance, posture correction, action correction
Referências
Kevin Adistambha, Christian H. Ritz, and Ian S. Burnett. 2008. Motion classification using Dynamic Time Warping. In 2008 IEEE 10th Workshop on Multimedia Signal Processing. 622–627. https://doi.org/10.1109/MMSP.2008.4665151
Talal Alatiah and Chen Chen. 2020. Recognizing Exercises and Counting Repetitions in Real Time. https://doi.org/10.48550/ARXIV.2005.03194
Federico Angelini, Zeyu Fu, Yang Long, Ling Shao, and Syed Mohsen Naqvi. 2018. ActionXPose: A Novel 2D Multi-view Pose-based Algorithm for Real-time Human Action Recognition. https://doi.org/10.48550/ARXIV.1810.12126
Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. 2020. Blazepose: On-device real-time body pose tracking. arXiv preprint arXiv:2006.10204 (2020).
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime MultiPerson 2D Pose Estimation Using Part Affinity Fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Ajay Chaudhari, Omkar Dalvi, Onkar Ramade, and Dayanand Ambawade. 2021. Yog-Guru: Real-Time Yoga Pose Correction System Using Deep Learning Methods. In 2021 International Conference on Communication information and Computing Technology (ICCICT). 1–6. https://doi.org/10.1109/ICCICT50803.2021.9509937
Steven Chen and Richard R. Yang. 2020. Pose Trainer: Correcting Exercise Posture using Pose Estimation. https://doi.org/10.48550/ARXIV.2006.11718
Weiming Chen, Zijie Jiang, Hailin Guo, and Xiaoyang Ni. 2020. Fall Detection Based on Key Points of Human-Skeleton Using OpenPose. Symmetry 12, 5 (2020). https://doi.org/10.3390/sym12050744
Gisela Miranda Difini, Marcio Garcia Martins, and Jorge Luis Victória Barbosa. 2021. Human Pose Estimation for Training Assistance: A Systematic Literature Review. Association for Computing Machinery, New York, NY, USA, 189–196. https://doi.org/10.1145/3470482.3479633
Felix Escalona, Ester Martinez-Martin, Edmanuel Cruz, Miguel Cazorla, and Francisco Gomez-Donoso. 2019. EVA: EVAluating at-home rehabilitation exercises using augmented reality and low-cost sensors. Virtual Reality (2019), 1–15. https://doi.org/10.1007/s10055-019-00419-4
Hong Guo, Shanchen Zou, Chuying Lai, and Hongxin Zhang. 2021. PhyCoVIS: A visual analytic tool of physical coordination for cheer and dance training. Computer Animation and Virtual Worlds 32, 1 (2021), e1975. https://doi.org/10.1002/cav.1975 arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cav.1975
Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, and Stephen Linkman. 2009. Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology 51, 1 (2009), 7–15. Special Section - MostCited Articles in 2002 and Regular Research Papers. https://doi.org/10.1016/j.infsof.2008.09.009
Jen Jui Liu, Jacob Newman, and Dah-Jye Lee. 2020. Body Motion Analysis for Golf Swing Evaluation. In Advances in Visual Computing, George Bebis, Zhaozheng Yin, Edward Kim, Jan Bender, Kartic Subr, Bum Chul Kwon, Jian Zhao, Denis Kalkofen, and George Baciu (Eds.). Springer International Publishing, Cham, 566–577. https://doi.org/10.1007/978-3-030-64556-4_44
Mandy Lu, Kathleen Poston, Adolf Pfefferbaum, Edith V. Sullivan, Li Fei-Fei, Kilian M. Pohl, Juan Carlos Niebles, and Ehsan Adeli. 2020. Vision-Based Estimation of MDS-UPDRS Gait Scores for Assessing Parkinson’s Disease Motor Severity. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Anne L. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoceanu, and Leo Joskowicz (Eds.). Springer International Publishing, Cham, 637–647. https://doi.org/10.1007/978-3-03059716-0_61
Cristian Militaru, Maria-Denisa Militaru, and Kuderna-Iulian Benta. 2020. Physical Exercise Form Correction Using Neural Networks. In Companion Publication of the 2020 International Conference on Multimodal Interaction (Virtual Event, Netherlands) (ICMI ’20 Companion). Association for Computing Machinery, New York, NY, USA, 240–244. https://doi.org/10.1145/3395035.3425302
Lene Nielsen. 2013. Personas - User Focused Design. Vol. 1373. Springer. https://doi.org/10.1007/978-1-4471-7427-1
Ashish Ohri, Shashank Agrawal, and Garima S Chaudhary. 2021. On-device Realtime Pose Estimation & Correction. (2021).
Santiago Riofrio, David Pozo, Jorge Rosero, and Juan Vasquez. 2017. Gesture Recognition Using Dynamic Time Warping and Kinect: A Practical Approach. 302–308. https://doi.org/10.1109/INCISCOS.2017.36
Anima Mishra Sharma, KS Venkatesh, and Amitabha Mukerjee. 2011. Human pose estimation in surveillance videos using temporal continuity on static pose. In 2011 International Conference on Image Information Processing. 1–6. https://doi.org/10.1109/ICIIP.2011.6108913
Dan Shi and Xin Jiang. 2021. Sport training action correction by using convolutional neural network. Internet Technology Letters 4, 3 (2021), e261. arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1002/itl2.261 https://doi.org/10.1002/itl2.261
Nicky Terblanche and Martin Kidd. 2022. Adoption Factors and Mod- erating Effects of Age and Gender That Influence the Intention to Use a Non-Directive Reflective Coaching Chatbot. SAGE Open 12, 2 (2022), 21582440221096136. https://doi.org/10.1177/21582440221096136
Maybel Chan Thar, Khine Zar Ne Winn, and Nobuo Funabiki. 2019. A Proposal of Yoga Pose Assessment Method Using Pose Detection for Self-Learning. In 2019 International Conference on Advanced Information Technologies (ICAIT). 137–142. https://doi.org/10.1109/AITC.2019.8920892
Viswanath Venkatesh, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003. User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly 27, 3 (2003), 425–478. https://doi.org/10.2307/30036540
Jianbo Wang, Kai Qiu, Houwen Peng, Jianlong Fu, and Jianke Zhu. 2019. AI Coach: Deep Human Pose Estimation and Analysis for Personalized Athletic Training Assistance. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). Association for Computing Machinery, New York, NY, USA, 374–382. https://doi.org/10.1145/3343031.3350910
Chung-Yi Weng, Brian Curless, and Ira Kemelmacher-Shlizerman. 2019. Photo Wake-Up: 3D Character Animation From a Single Photo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Barry D. Wilson. 2008. Development in video technology for coaching. Sports Technology 1, 1 (2008), 34–40. https://doi.org/10.1080/19346182.2008.9648449
Lei Yang, Yingxiang Li, Degui Zeng, and Dong Wang. 2021. Human Exercise Posture Analysis based on Pose Estimation. In 2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Vol. 5. 1715– 1719. https://doi.org/10.1109/IAEAC50856.2021.9390870
Haotian Zhang, Cristobal Sciutto, Maneesh Agrawala, and Kayvon Fatahalian. 2021. Vid2Player: Controllable Video Sprites That Behave and Appear Like Professional Tennis Players. ACM Trans. Graph. 40, 3, Article 24 (may 2021), 16 pages. https://doi.org/10.1145/3448978
Talal Alatiah and Chen Chen. 2020. Recognizing Exercises and Counting Repetitions in Real Time. https://doi.org/10.48550/ARXIV.2005.03194
Federico Angelini, Zeyu Fu, Yang Long, Ling Shao, and Syed Mohsen Naqvi. 2018. ActionXPose: A Novel 2D Multi-view Pose-based Algorithm for Real-time Human Action Recognition. https://doi.org/10.48550/ARXIV.1810.12126
Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. 2020. Blazepose: On-device real-time body pose tracking. arXiv preprint arXiv:2006.10204 (2020).
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime MultiPerson 2D Pose Estimation Using Part Affinity Fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Ajay Chaudhari, Omkar Dalvi, Onkar Ramade, and Dayanand Ambawade. 2021. Yog-Guru: Real-Time Yoga Pose Correction System Using Deep Learning Methods. In 2021 International Conference on Communication information and Computing Technology (ICCICT). 1–6. https://doi.org/10.1109/ICCICT50803.2021.9509937
Steven Chen and Richard R. Yang. 2020. Pose Trainer: Correcting Exercise Posture using Pose Estimation. https://doi.org/10.48550/ARXIV.2006.11718
Weiming Chen, Zijie Jiang, Hailin Guo, and Xiaoyang Ni. 2020. Fall Detection Based on Key Points of Human-Skeleton Using OpenPose. Symmetry 12, 5 (2020). https://doi.org/10.3390/sym12050744
Gisela Miranda Difini, Marcio Garcia Martins, and Jorge Luis Victória Barbosa. 2021. Human Pose Estimation for Training Assistance: A Systematic Literature Review. Association for Computing Machinery, New York, NY, USA, 189–196. https://doi.org/10.1145/3470482.3479633
Felix Escalona, Ester Martinez-Martin, Edmanuel Cruz, Miguel Cazorla, and Francisco Gomez-Donoso. 2019. EVA: EVAluating at-home rehabilitation exercises using augmented reality and low-cost sensors. Virtual Reality (2019), 1–15. https://doi.org/10.1007/s10055-019-00419-4
Hong Guo, Shanchen Zou, Chuying Lai, and Hongxin Zhang. 2021. PhyCoVIS: A visual analytic tool of physical coordination for cheer and dance training. Computer Animation and Virtual Worlds 32, 1 (2021), e1975. https://doi.org/10.1002/cav.1975 arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cav.1975
Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, and Stephen Linkman. 2009. Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology 51, 1 (2009), 7–15. Special Section - MostCited Articles in 2002 and Regular Research Papers. https://doi.org/10.1016/j.infsof.2008.09.009
Jen Jui Liu, Jacob Newman, and Dah-Jye Lee. 2020. Body Motion Analysis for Golf Swing Evaluation. In Advances in Visual Computing, George Bebis, Zhaozheng Yin, Edward Kim, Jan Bender, Kartic Subr, Bum Chul Kwon, Jian Zhao, Denis Kalkofen, and George Baciu (Eds.). Springer International Publishing, Cham, 566–577. https://doi.org/10.1007/978-3-030-64556-4_44
Mandy Lu, Kathleen Poston, Adolf Pfefferbaum, Edith V. Sullivan, Li Fei-Fei, Kilian M. Pohl, Juan Carlos Niebles, and Ehsan Adeli. 2020. Vision-Based Estimation of MDS-UPDRS Gait Scores for Assessing Parkinson’s Disease Motor Severity. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Anne L. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoceanu, and Leo Joskowicz (Eds.). Springer International Publishing, Cham, 637–647. https://doi.org/10.1007/978-3-03059716-0_61
Cristian Militaru, Maria-Denisa Militaru, and Kuderna-Iulian Benta. 2020. Physical Exercise Form Correction Using Neural Networks. In Companion Publication of the 2020 International Conference on Multimodal Interaction (Virtual Event, Netherlands) (ICMI ’20 Companion). Association for Computing Machinery, New York, NY, USA, 240–244. https://doi.org/10.1145/3395035.3425302
Lene Nielsen. 2013. Personas - User Focused Design. Vol. 1373. Springer. https://doi.org/10.1007/978-1-4471-7427-1
Ashish Ohri, Shashank Agrawal, and Garima S Chaudhary. 2021. On-device Realtime Pose Estimation & Correction. (2021).
Santiago Riofrio, David Pozo, Jorge Rosero, and Juan Vasquez. 2017. Gesture Recognition Using Dynamic Time Warping and Kinect: A Practical Approach. 302–308. https://doi.org/10.1109/INCISCOS.2017.36
Anima Mishra Sharma, KS Venkatesh, and Amitabha Mukerjee. 2011. Human pose estimation in surveillance videos using temporal continuity on static pose. In 2011 International Conference on Image Information Processing. 1–6. https://doi.org/10.1109/ICIIP.2011.6108913
Dan Shi and Xin Jiang. 2021. Sport training action correction by using convolutional neural network. Internet Technology Letters 4, 3 (2021), e261. arXiv: https://onlinelibrary.wiley.com/doi/pdf/10.1002/itl2.261 https://doi.org/10.1002/itl2.261
Nicky Terblanche and Martin Kidd. 2022. Adoption Factors and Mod- erating Effects of Age and Gender That Influence the Intention to Use a Non-Directive Reflective Coaching Chatbot. SAGE Open 12, 2 (2022), 21582440221096136. https://doi.org/10.1177/21582440221096136
Maybel Chan Thar, Khine Zar Ne Winn, and Nobuo Funabiki. 2019. A Proposal of Yoga Pose Assessment Method Using Pose Detection for Self-Learning. In 2019 International Conference on Advanced Information Technologies (ICAIT). 137–142. https://doi.org/10.1109/AITC.2019.8920892
Viswanath Venkatesh, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003. User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly 27, 3 (2003), 425–478. https://doi.org/10.2307/30036540
Jianbo Wang, Kai Qiu, Houwen Peng, Jianlong Fu, and Jianke Zhu. 2019. AI Coach: Deep Human Pose Estimation and Analysis for Personalized Athletic Training Assistance. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). Association for Computing Machinery, New York, NY, USA, 374–382. https://doi.org/10.1145/3343031.3350910
Chung-Yi Weng, Brian Curless, and Ira Kemelmacher-Shlizerman. 2019. Photo Wake-Up: 3D Character Animation From a Single Photo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Barry D. Wilson. 2008. Development in video technology for coaching. Sports Technology 1, 1 (2008), 34–40. https://doi.org/10.1080/19346182.2008.9648449
Lei Yang, Yingxiang Li, Degui Zeng, and Dong Wang. 2021. Human Exercise Posture Analysis based on Pose Estimation. In 2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Vol. 5. 1715– 1719. https://doi.org/10.1109/IAEAC50856.2021.9390870
Haotian Zhang, Cristobal Sciutto, Maneesh Agrawala, and Kayvon Fatahalian. 2021. Vid2Player: Controllable Video Sprites That Behave and Appear Like Professional Tennis Players. ACM Trans. Graph. 40, 3, Article 24 (may 2021), 16 pages. https://doi.org/10.1145/3448978
Publicado
07/11/2022
Como Citar
DIFINI, Gisela Miranda; MARTINS, Márcio Garcia; BARBOSA, Jorge Luis Victória.
A Movement Analysis Application using Human Pose Estimation and Action Correction. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 28. , 2022, Curitiba.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 385-393.
