Exploring Interaction in a Virtual Music Studio through Gesture Recognition on Smartwatches and HMD Devices
Resumo
This work proposes the development of an innovative virtual environment that simulates a music studio, exploring interaction through gesture recognition on smartwatches and mobile virtual reality devices (mVR). By utilizing Multiple Dimensional Dynamic Time Warping (MDDTW) for gesture recognition, users will be able to interact intuitively and immersively with virtual musical instruments. The smartwatch acts as an input device, capturing user gestures and transmitting them to the virtual environment. Moreover, the interaction also allows for the modification of musical samples and the visualization of instrument details. The combination of smartwatches and mVR devices offers a unique experience, expanding possibilities for expression and control for users. It is expected that this approach will drive human-computer interaction in various domains, showcasing new paths for the integrated use of established technologies in the market. The study aims to provide an engaging and immersive experience in the musical context, catering to both music enthusiasts and professionals, while contributing to the ongoing evolution of technological interaction.
Palavras-chave:
Smartwatch, HMD devices, gesture recognition, virtual reality, interaction
Referências
Eugenie Brasier, Olivier Chapuis, Nicolas Ferey, Jeanne Vezien, and Caroline Appert. 2020. ARPads: Mid-air Indirect Input for Augmented Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 332–343. https://doi.org/10.1109/ISMAR50242.2020.00060
Ruei-Che Chang, Chi-Huan Chiang, Shuo-wen Hsu, Chih-Yun Yang, Da-Yuan Huang, and Bing-Yu Chen. 2020. TanGo: Exploring Expressive Tangible Interactions on Head-Mounted Displays. In Proceedings of the 2020 ACM Symposium on Spatial User Interaction (Virtual Event, Canada) (SUI ’20). Association for Computing Machinery, New York, NY, USA, Article 3, 10 pages. https://doi.org/10.1145/3385959.3418457
Taizhou Chen, Lantian Xu, Xianshan Xu, and Kening Zhu. 2021. GestOnHMD: Enabling Gesture-based Interaction on Low-cost VR Head-Mounted Display. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021). https://doi.org/10.1109/TVCG.2021.3067689
Augusto Esteves, Yonghwan Shin, and Ian Oakley. 2020. Comparing selection mechanisms for gaze input techniques in head-mounted displays. International Journal of Human-Computer Studies 139 (2020), 102414. https://doi.org/10.1016/j.ijhcs.2020.102414
Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: Smartwatch-Based 3D Pointing for Smartphone-Based Virtual Reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST ’16). Association for Computing Machinery, New York, NY, USA.
Mateus M. Luna, Thyago P. Carvalho, Fabrizzio Alphonsus A. M. N. Soares, Hugo A. D. Nascimento, and Ronaldo M. Costa. 2017. Wrist Player: A Smartwatch Gesture Controller for Smart TVs. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. 336–341. https://doi.org/10.1109/COMPSAC.2017.266
Yuqi Luo, Jiang Liu, and Shigeru Shimamoto. 2021. Wearable Air-Writing Recognition System employing Dynamic Time Warping. In 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). 1–6. https://doi.org/10.1109/CCNC49032.2021.9369458
Samridhi Mathur and Poonam Sharma. 2018. Sign Language Gesture Recognition using Zernike Moments and DTW. In 2018 5th International Conference on Signal Processing and Integrated Networks (SPIN). 586–591. https://doi.org/10.1109/SPIN.2018.8474179
John McCaffery, Alan Miller, Anna Vermehren, and Adeola Fabola. 2015. The Virtual Museums of Caen: A case study on modes of representation of digital historical content. In 2015 Digital Heritage, Vol. 2. 541–548. https://doi.org/10.1109/DigitalHeritage.2015.7419571
Danial Moazen, Seyed A Sajjadi, and Ani Nahapetian. 2016. AirDraw: Leveraging smart watch motion sensors for mobile human computer interactions. In 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC). 442–446. https://doi.org/10.1109/CCNC.2016.7444820
Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815
T. H. Nascimento, C. B. R. Ferreira, Wellington G. Rodrigues, and Fabrizzio Soares. 2020. Interaction with Smartwatches Using Gesture Recognition: A Systematic Literature Review. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). 1661–1666. https://doi.org/10.1109/COMPSAC48688.2020.00-17
Thamer Horbylon Nascimento, Fabrizzio Soares, Hugo Nascimento, Rogerio Salvini, Mateus Luna, Cristhiane Gonçalves, and Eduardo Souza. 2018. Interaction with Platform Games Using Smartwatches and Continuous Gesture Recognition: A Case Study. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 02. 253–258. https://doi.org/10.1109/COMPSAC.2018.10239
Thamer Horbylon Nascimento, Fabrizzio Soares, Hugo Nascimento, Marcos Vieira, Thyago Carvalho, and Wesley Miranda. 2019. Netflix Control Method Using Smartwatches and Continuous Gesture Recognition. In 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE). 1–4. https://doi.org/10.1109/CCECE.2019.8861610
Thamer Horbylon Nascimento, Fabrizzio Soares, Danilo Vieira Oliveira, Rogerio Lopes Salvini, Ronaldo Martins da Costa, and Cristhiane Gonçalves. 2017. Method for Text Input with Google Cardboard: An Approach Using Smartwatches and Continuous Gesture Recognition. In 2017 19th Symposium on Virtual and Augmented Reality (SVR). 223–226. https://doi.org/10.1109/SVR.2017.36
Sang-Min Park and Young-Gab Kim. 2022. A Metaverse: Taxonomy, Components, Applications, and Open Challenges. IEEE Access 10 (2022), 4209–4251. https://doi.org/10.1109/ACCESS.2021.3140175
I. Picallo, A. Vidal-Balea, O. Blanco-Novoa, P. Lopez-Iturri, P. Fraga-Lamas, H. Klaina, T. M. Fernández-Caramés, L. Azpilicueta, and F. Falcone. 2021. Design and Experimental Validation of an Augmented Reality System With Wireless Integration for Context Aware Enhanced Show Experience in Auditoriums. IEEE Access 9 (2021), 5466–5484. https://doi.org/10.1109/ACCESS.2020.3048203
Xiaogang Ruan and Chongyang Tian. 2015. Dynamic gesture recognition based on improved DTW algorithm. In 2015 IEEE International Conference on Mechatronics and Automation (ICMA). 2134–2138. https://doi.org/10.1109/ICMA.2015.7237816
Namrata Singh and Sarvpal Singh. 2017. Virtual reality: A brief survey. In 2017 International Conference on Information Communication and Embedded Systems (ICICES). 1–6. https://doi.org/10.1109/ICICES.2017.8070720
Gineke A Ten Holt, Marcel JT Reinders, and Emile A Hendriks. 2007. Multidimensional dynamic time warping for gesture recognition. In Thirteenth annual conference of the Advanced School for Computing and Imaging, Vol. 300. 1.
Pui Chung Wong, Kening Zhu, Xing-Dong Yang, and Hongbo Fu. 2020. Exploring Eyes-Free Bezel-Initiated Swipe on Round Smartwatches. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376393
Wenge Xu, Hai-Ning Liang, Yuxuan Zhao, Tianyu Zhang, Difeng Yu, and Diego Monteiro. 2019. RingText: Dwell-free and hands-free Text Entry for Mobile Head-Mounted Displays using Head Motions. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1991–2001. https://doi.org/10.1109/TVCG.2019.2898736
Huixin Zhang, Ya Hu, Jun Zhu, Lin Fu, Bingli Xu, and Weilian Li. 2022. A gazebased interaction method for large-scale and large-space disaster scenes within mobile virtual reality. Transactions in GIS 26, 3 (2022).
Yuzhou Zhuang, Yuntao Wang, Yukang Yan, Xuhai Xu, and Yuanchun Shi. 2021. ReflecTrack: Enabling 3D Acoustic Position Tracking Using Commodity Dual-Microphone Smartphones. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 1050–1062. https://doi.org/10.1145/3472749.3474805
Ruei-Che Chang, Chi-Huan Chiang, Shuo-wen Hsu, Chih-Yun Yang, Da-Yuan Huang, and Bing-Yu Chen. 2020. TanGo: Exploring Expressive Tangible Interactions on Head-Mounted Displays. In Proceedings of the 2020 ACM Symposium on Spatial User Interaction (Virtual Event, Canada) (SUI ’20). Association for Computing Machinery, New York, NY, USA, Article 3, 10 pages. https://doi.org/10.1145/3385959.3418457
Taizhou Chen, Lantian Xu, Xianshan Xu, and Kening Zhu. 2021. GestOnHMD: Enabling Gesture-based Interaction on Low-cost VR Head-Mounted Display. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021). https://doi.org/10.1109/TVCG.2021.3067689
Augusto Esteves, Yonghwan Shin, and Ian Oakley. 2020. Comparing selection mechanisms for gaze input techniques in head-mounted displays. International Journal of Human-Computer Studies 139 (2020), 102414. https://doi.org/10.1016/j.ijhcs.2020.102414
Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: Smartwatch-Based 3D Pointing for Smartphone-Based Virtual Reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST ’16). Association for Computing Machinery, New York, NY, USA.
Mateus M. Luna, Thyago P. Carvalho, Fabrizzio Alphonsus A. M. N. Soares, Hugo A. D. Nascimento, and Ronaldo M. Costa. 2017. Wrist Player: A Smartwatch Gesture Controller for Smart TVs. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. 336–341. https://doi.org/10.1109/COMPSAC.2017.266
Yuqi Luo, Jiang Liu, and Shigeru Shimamoto. 2021. Wearable Air-Writing Recognition System employing Dynamic Time Warping. In 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). 1–6. https://doi.org/10.1109/CCNC49032.2021.9369458
Samridhi Mathur and Poonam Sharma. 2018. Sign Language Gesture Recognition using Zernike Moments and DTW. In 2018 5th International Conference on Signal Processing and Integrated Networks (SPIN). 586–591. https://doi.org/10.1109/SPIN.2018.8474179
John McCaffery, Alan Miller, Anna Vermehren, and Adeola Fabola. 2015. The Virtual Museums of Caen: A case study on modes of representation of digital historical content. In 2015 Digital Heritage, Vol. 2. 541–548. https://doi.org/10.1109/DigitalHeritage.2015.7419571
Danial Moazen, Seyed A Sajjadi, and Ani Nahapetian. 2016. AirDraw: Leveraging smart watch motion sensors for mobile human computer interactions. In 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC). 442–446. https://doi.org/10.1109/CCNC.2016.7444820
Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815
T. H. Nascimento, C. B. R. Ferreira, Wellington G. Rodrigues, and Fabrizzio Soares. 2020. Interaction with Smartwatches Using Gesture Recognition: A Systematic Literature Review. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). 1661–1666. https://doi.org/10.1109/COMPSAC48688.2020.00-17
Thamer Horbylon Nascimento, Fabrizzio Soares, Hugo Nascimento, Rogerio Salvini, Mateus Luna, Cristhiane Gonçalves, and Eduardo Souza. 2018. Interaction with Platform Games Using Smartwatches and Continuous Gesture Recognition: A Case Study. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 02. 253–258. https://doi.org/10.1109/COMPSAC.2018.10239
Thamer Horbylon Nascimento, Fabrizzio Soares, Hugo Nascimento, Marcos Vieira, Thyago Carvalho, and Wesley Miranda. 2019. Netflix Control Method Using Smartwatches and Continuous Gesture Recognition. In 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE). 1–4. https://doi.org/10.1109/CCECE.2019.8861610
Thamer Horbylon Nascimento, Fabrizzio Soares, Danilo Vieira Oliveira, Rogerio Lopes Salvini, Ronaldo Martins da Costa, and Cristhiane Gonçalves. 2017. Method for Text Input with Google Cardboard: An Approach Using Smartwatches and Continuous Gesture Recognition. In 2017 19th Symposium on Virtual and Augmented Reality (SVR). 223–226. https://doi.org/10.1109/SVR.2017.36
Sang-Min Park and Young-Gab Kim. 2022. A Metaverse: Taxonomy, Components, Applications, and Open Challenges. IEEE Access 10 (2022), 4209–4251. https://doi.org/10.1109/ACCESS.2021.3140175
I. Picallo, A. Vidal-Balea, O. Blanco-Novoa, P. Lopez-Iturri, P. Fraga-Lamas, H. Klaina, T. M. Fernández-Caramés, L. Azpilicueta, and F. Falcone. 2021. Design and Experimental Validation of an Augmented Reality System With Wireless Integration for Context Aware Enhanced Show Experience in Auditoriums. IEEE Access 9 (2021), 5466–5484. https://doi.org/10.1109/ACCESS.2020.3048203
Xiaogang Ruan and Chongyang Tian. 2015. Dynamic gesture recognition based on improved DTW algorithm. In 2015 IEEE International Conference on Mechatronics and Automation (ICMA). 2134–2138. https://doi.org/10.1109/ICMA.2015.7237816
Namrata Singh and Sarvpal Singh. 2017. Virtual reality: A brief survey. In 2017 International Conference on Information Communication and Embedded Systems (ICICES). 1–6. https://doi.org/10.1109/ICICES.2017.8070720
Gineke A Ten Holt, Marcel JT Reinders, and Emile A Hendriks. 2007. Multidimensional dynamic time warping for gesture recognition. In Thirteenth annual conference of the Advanced School for Computing and Imaging, Vol. 300. 1.
Pui Chung Wong, Kening Zhu, Xing-Dong Yang, and Hongbo Fu. 2020. Exploring Eyes-Free Bezel-Initiated Swipe on Round Smartwatches. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376393
Wenge Xu, Hai-Ning Liang, Yuxuan Zhao, Tianyu Zhang, Difeng Yu, and Diego Monteiro. 2019. RingText: Dwell-free and hands-free Text Entry for Mobile Head-Mounted Displays using Head Motions. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1991–2001. https://doi.org/10.1109/TVCG.2019.2898736
Huixin Zhang, Ya Hu, Jun Zhu, Lin Fu, Bingli Xu, and Weilian Li. 2022. A gazebased interaction method for large-scale and large-space disaster scenes within mobile virtual reality. Transactions in GIS 26, 3 (2022).
Yuzhou Zhuang, Yuntao Wang, Yukang Yan, Xuhai Xu, and Yuanchun Shi. 2021. ReflecTrack: Enabling 3D Acoustic Position Tracking Using Commodity Dual-Microphone Smartphones. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 1050–1062. https://doi.org/10.1145/3472749.3474805
Publicado
06/11/2023
Como Citar
DE CASTRO, Murilo Santos; GOMES, Pedro Raphael Inácio; NASCIMENTO, Thamer Horbylon.
Exploring Interaction in a Virtual Music Studio through Gesture Recognition on Smartwatches and HMD Devices. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 25. , 2023, Rio Grande/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 284–288.
