Gesture Recognition in Smartwatches Using LSTM for Interaction in Low-Cost Virtual Environments
Resumo
This work presents a methodology to enhance interaction in VR environments using accessible devices such as Google Cardboard and smartwatches. The main contribution is the implementation of a recurrent neural network model of the LSTM type, designed to recognize gestures captured by smartwatches. This enables more natural and fluid interaction with the virtual environment, significantly elevating the level of immersion and responsiveness perceived by users. Preliminary results demonstrate that the LSTM model achieves robust performance in accurately identifying gestures, which is essential for providing an immersive and engaging experience. We hope that this approach expands the possibilities for interaction in virtual environments and represents a significant advancement in the field of wearable computing applied to Virtual Reality.
Palavras-chave:
Gesture Recognition, Smartwatches, Virtual Reality, LSTM Algorithm
Referências
John Brooke. 1996. SUS: A 'Quick and Dirty' Usability Scale. In Usability Evaluation In Industry (first ed.). Taylor and Francis, London, 189–194.
Phillipe Valente Cardoso and Kairo da Silva Santos. 2015. Realidade virtual e geografia: o caso do Google cardboard glasses para o ensino. Revista Tamoios 11, 2 (2015).
Murilo Santos de Castro, Pedro Raphael Inácio Gomes, and Thamer Horbylon Nascimento. 2024. Exploring Interaction in a Virtual Music Studio through Gesture Recognition on Smartwatches and HMD Devices. In Proceedings of the 25th Symposium on Virtual and Augmented Reality (Rio Grande, Brazil) (SVR ’23). Association for Computing Machinery, New York, NY, USA, 284–288. DOI: 10.1145/3625008.3625049
Yufeng Deng, Dong Wang, Qian Zhang, Run Zhao, and Bo Chen. 2018. ReaderTrack: Reader-Book Interaction Reasoning Using RFID and Smartwatch. In 2018 27th International Conference on Computer Communication and Networks (ICCCN). IEEE, 1–9.
Pedro Raphael Inácio Gomes, Murillo Santos de Castro, and Thamer Horbylon Nascimento. 2024. Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature Review. In Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems (, Maceió, Brazil,) (IHC ’23). Association for Computing Machinery, New York, NY, USA, Article 55, 11 pages. DOI: 10.1145/3638067.3638082
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183.
Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Jussi PP Jokinen, Tuuli Keskinen, and Roope Raisamo. 2013. Designing gesture-based control for factory automation. In Human-Computer Interaction–INTERACT 2013: 14th IFIP TC 13 International Conference, Cape Town, South Africa, September 2-6, 2013, Proceedings, Part II 14. Springer, 202–209.
Thamer Horbylon Nascimento, Fabrizzio Alphonsus A.M.N. Soares, Hugo A. D. Nascimento, Rogerio L. Salvini, Mateus M. Luna, Cristhiane Gonçalves, and Eduardo F. Souza. 2018. Interaction with Platform Games Using Smartwatches and Continuous Gesture Recognition: A Case Study. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 02. 253–258. DOI: 10.1109/COMPSAC.2018.10239
Mauri Kaipainen, Niklas Ravaja, Pia Tikka, Rasmus Vuori, Roberto Pugliese, Marco Rapino, and Tapio Takala. 2011. Enactive Systems and Enactive Media: Embodied Human-Machine Coupling beyond Interfaces. Leonardo 44, 5 (10 2011), 433–438. DOI: 10.1162/LEON_a_00244
Varun Badrinath Krishna. 2020. Ballroom dance movement recognition using a Smart Watch. arXiv preprint arXiv:2008.10122 (2020).
Ho Chul Lee and Dong Myung Lee. 2020. Machine learning model for indoor localization algorithm using GPS and smart watch. In 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC). IEEE, 735–737.
Gen Li, Lingfeng Zhang, and Hiroyuki Sato. 2021. In-air signature authentication using smartwatch motion sensors. In 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 386–395.
Yande Li, Taiqian Wang, Lian Li, Caihong Li, Yi Yang, Li Liu, et al. 2018. Hand gesture recognition and real-time game control based on a wearable band with 6-axis sensors. In 2018 international joint conference on neural networks (IJCNN). IEEE, 1–6.
Carlos Marín-Lora, Miguel Chover, Micaela Yanet Martín, and Linda García-Rytman. 2023. Creating a treadmill running video game with smartwatch interaction. Multimedia Tools and Applications (2023), 1–21.
Thamer Horbylon Nascimento, Deborah Fernandes, Gabriel Vieira, Juliana Felix, Murillo Castro, and Fabrizzio Soares. 2023. MazeVR: Immersion and Interaction Using Google Cardboard and Continuous Gesture Recognition on Smartwatches. In Proceedings of the 28th International ACM Conference on 3D Web Technology (San Sebastian, Spain) (Web3D ’23). Association for Computing Machinery, New York, NY, USA, Article 18, 5 pages. DOI: 10.1145/3611314.3615912
T. H. Nascimento, C. B. R. Ferreira, Wellington G. Rodrigues, and Fabrizzio Soares. 2020. Interaction with Smartwatches Using Gesture Recognition: A Systematic Literature Review. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). 1661–1666. DOI: 10.1109/COMPSAC48688.2020.00-17
Thamer Horbylon Nascimento, Fabrizzio Alphonsus A. M. Nunes Soares, Danilo Vieira Oliveira, Rogerio Lopes Salvini, Ronaldo Martins da Costa, and Cristhiane Gonçalves. 2017. Method for Text Input with Google Cardboard: An Approach Using Smartwatches and Continuous Gesture Recognition. In 2017 19th Symposium on Virtual and Augmented Reality (SVR). 223–226. DOI: 10.1109/SVR.2017.36
Tatiana Teixeira Silveira Neiva. 2023. Realidade Virtual e Geografia: o uso do CardBoard Glasses. Revista Educação Geográfica em Foco 7, 13 (2023).
Marcus Vinicius O Nunes and Alexandre Cardoso. 2021. O uso de jogos de realidade virtual para o ensino de Computaçao Gráfica. In SBC – Proceedings of SBGames 2021. SBC.
Zilin Wang and Moon-Tong Chan. 2024. A systematic review of google cardboard used in education. Computers & Education: X Reality 4 (2024), 100046.
Anthony D Whitehead. 2014. Gesture Recognition with Accelerometers for Game Controllers, Phones and Wearables. GSTF Journal on Computing (JoC) 3 (2014), 1–7.
Yidi Zhang, Mengyuan Ran, Jun Liao, Guoxin Su, Ming Liu, and Li Liu. 2021. Stacked LSTM-Based Dynamic Hand Gesture Recognition with Six-Axis Motion Sensors. In 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2568–2575.
Phillipe Valente Cardoso and Kairo da Silva Santos. 2015. Realidade virtual e geografia: o caso do Google cardboard glasses para o ensino. Revista Tamoios 11, 2 (2015).
Murilo Santos de Castro, Pedro Raphael Inácio Gomes, and Thamer Horbylon Nascimento. 2024. Exploring Interaction in a Virtual Music Studio through Gesture Recognition on Smartwatches and HMD Devices. In Proceedings of the 25th Symposium on Virtual and Augmented Reality (Rio Grande, Brazil) (SVR ’23). Association for Computing Machinery, New York, NY, USA, 284–288. DOI: 10.1145/3625008.3625049
Yufeng Deng, Dong Wang, Qian Zhang, Run Zhao, and Bo Chen. 2018. ReaderTrack: Reader-Book Interaction Reasoning Using RFID and Smartwatch. In 2018 27th International Conference on Computer Communication and Networks (ICCCN). IEEE, 1–9.
Pedro Raphael Inácio Gomes, Murillo Santos de Castro, and Thamer Horbylon Nascimento. 2024. Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature Review. In Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems (, Maceió, Brazil,) (IHC ’23). Association for Computing Machinery, New York, NY, USA, Article 55, 11 pages. DOI: 10.1145/3638067.3638082
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183.
Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Jussi PP Jokinen, Tuuli Keskinen, and Roope Raisamo. 2013. Designing gesture-based control for factory automation. In Human-Computer Interaction–INTERACT 2013: 14th IFIP TC 13 International Conference, Cape Town, South Africa, September 2-6, 2013, Proceedings, Part II 14. Springer, 202–209.
Thamer Horbylon Nascimento, Fabrizzio Alphonsus A.M.N. Soares, Hugo A. D. Nascimento, Rogerio L. Salvini, Mateus M. Luna, Cristhiane Gonçalves, and Eduardo F. Souza. 2018. Interaction with Platform Games Using Smartwatches and Continuous Gesture Recognition: A Case Study. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 02. 253–258. DOI: 10.1109/COMPSAC.2018.10239
Mauri Kaipainen, Niklas Ravaja, Pia Tikka, Rasmus Vuori, Roberto Pugliese, Marco Rapino, and Tapio Takala. 2011. Enactive Systems and Enactive Media: Embodied Human-Machine Coupling beyond Interfaces. Leonardo 44, 5 (10 2011), 433–438. DOI: 10.1162/LEON_a_00244
Varun Badrinath Krishna. 2020. Ballroom dance movement recognition using a Smart Watch. arXiv preprint arXiv:2008.10122 (2020).
Ho Chul Lee and Dong Myung Lee. 2020. Machine learning model for indoor localization algorithm using GPS and smart watch. In 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC). IEEE, 735–737.
Gen Li, Lingfeng Zhang, and Hiroyuki Sato. 2021. In-air signature authentication using smartwatch motion sensors. In 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 386–395.
Yande Li, Taiqian Wang, Lian Li, Caihong Li, Yi Yang, Li Liu, et al. 2018. Hand gesture recognition and real-time game control based on a wearable band with 6-axis sensors. In 2018 international joint conference on neural networks (IJCNN). IEEE, 1–6.
Carlos Marín-Lora, Miguel Chover, Micaela Yanet Martín, and Linda García-Rytman. 2023. Creating a treadmill running video game with smartwatch interaction. Multimedia Tools and Applications (2023), 1–21.
Thamer Horbylon Nascimento, Deborah Fernandes, Gabriel Vieira, Juliana Felix, Murillo Castro, and Fabrizzio Soares. 2023. MazeVR: Immersion and Interaction Using Google Cardboard and Continuous Gesture Recognition on Smartwatches. In Proceedings of the 28th International ACM Conference on 3D Web Technology (San Sebastian, Spain) (Web3D ’23). Association for Computing Machinery, New York, NY, USA, Article 18, 5 pages. DOI: 10.1145/3611314.3615912
T. H. Nascimento, C. B. R. Ferreira, Wellington G. Rodrigues, and Fabrizzio Soares. 2020. Interaction with Smartwatches Using Gesture Recognition: A Systematic Literature Review. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). 1661–1666. DOI: 10.1109/COMPSAC48688.2020.00-17
Thamer Horbylon Nascimento, Fabrizzio Alphonsus A. M. Nunes Soares, Danilo Vieira Oliveira, Rogerio Lopes Salvini, Ronaldo Martins da Costa, and Cristhiane Gonçalves. 2017. Method for Text Input with Google Cardboard: An Approach Using Smartwatches and Continuous Gesture Recognition. In 2017 19th Symposium on Virtual and Augmented Reality (SVR). 223–226. DOI: 10.1109/SVR.2017.36
Tatiana Teixeira Silveira Neiva. 2023. Realidade Virtual e Geografia: o uso do CardBoard Glasses. Revista Educação Geográfica em Foco 7, 13 (2023).
Marcus Vinicius O Nunes and Alexandre Cardoso. 2021. O uso de jogos de realidade virtual para o ensino de Computaçao Gráfica. In SBC – Proceedings of SBGames 2021. SBC.
Zilin Wang and Moon-Tong Chan. 2024. A systematic review of google cardboard used in education. Computers & Education: X Reality 4 (2024), 100046.
Anthony D Whitehead. 2014. Gesture Recognition with Accelerometers for Game Controllers, Phones and Wearables. GSTF Journal on Computing (JoC) 3 (2014), 1–7.
Yidi Zhang, Mengyuan Ran, Jun Liao, Guoxin Su, Ming Liu, and Li Liu. 2021. Stacked LSTM-Based Dynamic Hand Gesture Recognition with Six-Axis Motion Sensors. In 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2568–2575.
Publicado
30/09/2024
Como Citar
SILVA, Leonardo; SOARES, Fabrizzio; FELIX, Juliana; CARDOSO, Luciana; ARANHA, Renan Vinicius; NASCIMENTO, Thamer Horbylon.
Gesture Recognition in Smartwatches Using LSTM for Interaction in Low-Cost Virtual Environments. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 26. , 2024, Manaus/AM.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 284-288.
