Machine Learning Applied to Locomotion in Virtual Reality

• Fernando Kenji Sakabe Insper https://orcid.org/0009-0004-0437-4605
• Fabio José Ayres Insper https://orcid.org/0009-0000-6821-4687
• Luciano Pereira Soares Insper https://orcid.org/0000-0003-3187-4168

Resumo

The objective of this research project is to recognize a user's walking pattern on a treadmill-like platform for navigation in Virtual Reality (VR) environments. To achieve this, a walking recognition software solution was developed using Convolutional Neural Networks (CNNs), a type of Machine Learning (ML) architecture. The ML model was trained on a dataset collected from users' movements in a virtual simulation, tracking the positions and rotations of their feet as they walked in various directions and orientations. Tracking was accomplished using 6 degrees of freedom (6DoF) trackers placed on the users' feet. The neural network achieved a 94% accuracy rate during testing. Due to the network's lightweight configuration, the response time is, on average, 0.1 seconds, demonstrating its potential as an efficient natural controller for real-time user navigation in multiple directions.

Palavras-chave: Walking in Virtual Reality, Machine Learning, Feet Tracking, Virtual Locomotion, Walking Pattern Recognition, User Navigation

Referências

Nilsson, Niels Christian et al. 2018. Natural walking in virtual reality: A review. Computers in Entertainment (CIE), v. 16, n. 2, p. 1-22, 2018. Retrieved June 29, 2024 from DOI: 10.1145/3180658.

KAT VR. 2024. Los Angeles, 2024. Retrieved June 29, 2024 from [link].

KERAS. 2024. Mountain View, 2024. Retrieved June 29, 2024 from [link].

Boletsis, Costas; Cedergren, Jarl Erik. 2019. VR locomotion in the new era of virtual reality: an empirical comparison of prevalent techniques. Advances in Human-Computer Interaction, v. 2019, 2019. Retrieved June 29, 2024 from DOI: 10.1155/2019/7420781.

Templeman, James N.; Denbrook, Patricia S.; Sibert, Linda E. 1999. Virtual locomotion: Walking in place through virtual environments. Presence, v. 8, n. 6, p. 598-617, 1999. Retrieved June 29, 2024 from DOI: 10.1162/105474699566512.

Langbehn, Eike et al. 2017. Application of redirected walking in room-scale VR. In: 2017 IEEE Virtual Reality (VR). IEEE, 2017. p. 449-450. Retrieved June 29, 2024 from DOI: 10.1109/VR.2017.7892373.

Figueroa Jacinto, Rosemarie; Kappler, Elizabeth. 2022. A discussion on accessibility, inclusivity, cultural design considerations, and health & safety advisories for virtual reality head mounted displays. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Sage CA: Los Angeles, CA: SAGE Publications, 2022. p. 1706-1710. Retrieved June 29, 2024 from DOI: 10.1177/1071181322661200.

Boletsis, Costas. 2017. The new era of virtual reality locomotion: A systematic literature review of techniques and a proposed typology. Multimodal Technologies and Interaction, v. 1, n. 4, p. 24, 2017. Retrieved June 29, 2024 from DOI: 10.3390/mti1040024.

Wehden, Lars-Ole et al. 2021. The slippery path to total presence: How omnidirectional virtual reality treadmills influence the gaming experience. Media and Communication, v. 9, n. 1, p. 5-16, 2021. Retrieved June 29, 2024 from DOI: 10.17645/mac.v9i1.3170.

Lee, Jiwon; Kim, Mingyu; Kim, Jinmo. 2017. A study on immersion and VR sickness in walking interaction for immersive virtual reality applications. Symmetry, v. 9, n. 5, p. 78, 2017. Retrieved June 29, 2024 from DOI: 10.3390/sym9050078.

Day, Brian L.; Fitzpatrick, Richard C. 2005. The vestibular system. Current biology, v. 15, n. 15, p. R583-R586, 2005. Retrieved June 29, 2024 from DOI: 10.1016/j.cub.2005.07.053.

Nilsson, Niels Christian; Serafin, Stefania; Nordahl, Rolf. 2016. Walking in place through virtual worlds. In: Human-Computer Interaction. Interaction Platforms and Techniques: 18th International Conference, HCI International 2016, Toronto, ON, Canada, July 17-22, 2016. Proceedings, Part II 18. Springer International Publishing, 2016. p. 37-48. Retrieved June 29, 2024 from DOI: 10.1007/978-3-319-39516-6_4.

Courtney. 2022. 5 Best VR Treadmills. Retrieved June 29, 2024 from [link].

Zou, Qin et al. 2020. Deep learning-based gait recognition using smartphones in the wild. IEEE Transactions on Information Forensics and Security, v. 15, p. 3197-3212, 2020. Retrieved June 29, 2024 from DOI: 10.1109/TIFS.2020.2985628.

Seethi, Venkata Devesh Reddy; Bharti, Pratool. 2020. Cnn-based speed detection algorithm for walking and running using wrist-worn wearable sensors. In: 2020 IEEE International Conference on Smart Computing (SMARTCOMP). IEEE, 2020. p. 278-283. Retrieved June 29, 2024 from DOI: 10.48550/arXiv.2006.02348.

Ordóñez, Francisco Javier; Roggen, Daniel. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, v. 16, n. 1, p. 115, 2016. Retrieved June 29, 2024 from DOI: 10.3390/s16010115.

Mannini, Andrea; Sabatini, Angelo Maria. 2011. On-line classification of human activity and estimation of walk-run speed from acceleration data using support vector machines. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2011. p. 3302-3305. Retrieved June 29, 2024 from DOI: 10.1109/IEMBS.2011.6090896.

Caserman, Polona et al. 2019. Real-time body tracking in virtual reality using a Vive tracker. Virtual Reality, v. 23, p. 155-168, 2019. Retrieved June 29, 2024 from DOI: 10.1007/s10055-018-0374-z.

Guaitolini, Michelangelo et al. 2021. Evaluating the accuracy of virtual reality trackers for computing spatiotemporal gait parameters. Sensors, v. 21, n. 10, p. 3325, 2021. Retrieved June 29, 2024 from DOI: 10.3390/s21103325.

Wu, Rui et al. 2020. Evaluation of virtual reality tracking performance for indoor navigation. In: 2020 IEEE/ION Position, Location and Navigation Symposium (PLANS). IEEE, 2020. p. 1311-1316. Retrieved June 29, 2024 from DOI: 10.1109/PLANS46316.2020.9110225.

Felicia, Patrick. 2019. Unity from Zero to Proficiency (Beginner): A Step-by-step guide to coding your first game. Publisher Name. Independently published. ISBN-13: 978-1091872028.

Uvicorn. 2024. Uvicorn ASGI Server. Retrieved June 22, 2024 from [link].

FastAPI. 2023. FastAPI Web Framework. Retrieved June 22, 2024 from [link].

Bouillod, Anthony et al. 2016. Validity and reliability of the 3D motion analyzer in comparison with the Vicon device for biomechanical pedalling analysis. In: 4th International Congress on Sport Sciences Research and Technology Support (IcSPORTS). Retrieved June 22, 2024 from [link].

Matsuda, Y., Nakamura, J., Amemiya, T., Ikei, Y., & Kitazaki, M. 2021. Enhancing virtual walking sensation using Self-Avatar in First-Person perspective and foot vibrations. Frontiers in Virtual Reality, 2. Retrieved June 28, 2024 from DOI: 10.3389/frvir.2021.654088.