Murilo Santos de Castro
ABSTRACT
This work proposes the development of an innovative virtual environment that simulates a music studio, exploring interaction through gesture recognition on smartwatches and mobile virtual reality devices (mVR). By utilizing Multiple Dimensional Dynamic Time Warping (MDDTW) for gesture recognition, users will be able to interact intuitively and immersively with virtual musical instruments. The smartwatch acts as an input device, capturing user gestures and transmitting them to the virtual environment. Moreover, the interaction also allows for the modification of musical samples and the visualization of instrument details. The combination of smartwatches and mVR devices offers a unique experience, expanding possibilities for expression and control for users. It is expected that this approach will drive human-computer interaction in various domains, showcasing new paths for the integrated use of established technologies in the market. The study aims to provide an engaging and immersive experience in the musical context, catering to both music enthusiasts and professionals, while contributing to the ongoing evolution of technological interaction.
- Eugenie Brasier, Olivier Chapuis, Nicolas Ferey, Jeanne Vezien, and Caroline Appert. 2020. ARPads: Mid-air Indirect Input for Augmented Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 332–343. https://doi.org/10.1109/ISMAR50242.2020.00060Google Scholar
Cross Ref
- Ruei-Che Chang, Chi-Huan Chiang, Shuo-wen Hsu, Chih-Yun Yang, Da-Yuan Huang, and Bing-Yu Chen. 2020. TanGo: Exploring Expressive Tangible Interactions on Head-Mounted Displays. In Proceedings of the 2020 ACM Symposium on Spatial User Interaction (Virtual Event, Canada) (SUI ’20). Association for Computing Machinery, New York, NY, USA, Article 3, 10 pages. https://doi.org/10.1145/3385959.3418457Google ScholarDigital Library
- Taizhou Chen, Lantian Xu, Xianshan Xu, and Kening Zhu. 2021. GestOnHMD: Enabling Gesture-based Interaction on Low-cost VR Head-Mounted Display. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021). https://doi.org/10.1109/TVCG.2021.3067689Google ScholarCross Ref
- Augusto Esteves, Yonghwan Shin, and Ian Oakley. 2020. Comparing selection mechanisms for gaze input techniques in head-mounted displays. International Journal of Human-Computer Studies 139 (2020), 102414. https://doi.org/10.1016/j.ijhcs.2020.102414Google ScholarCross Ref
- Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: Smartwatch-Based 3D Pointing for Smartphone-Based Virtual Reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST ’16). Association for Computing Machinery, New York, NY, USA.Google ScholarDigital Library
- Mateus M. Luna, Thyago P. Carvalho, Fabrizzio Alphonsus A. M. N. Soares, Hugo A. D. Nascimento, and Ronaldo M. Costa. 2017. Wrist Player: A Smartwatch Gesture Controller for Smart TVs. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. 336–341. https://doi.org/10.1109/COMPSAC.2017.266Google ScholarCross Ref
- Yuqi Luo, Jiang Liu, and Shigeru Shimamoto. 2021. Wearable Air-Writing Recognition System employing Dynamic Time Warping. In 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). 1–6. https://doi.org/10.1109/CCNC49032.2021.9369458Google ScholarDigital Library
- Samridhi Mathur and Poonam Sharma. 2018. Sign Language Gesture Recognition using Zernike Moments and DTW. In 2018 5th International Conference on Signal Processing and Integrated Networks (SPIN). 586–591. https://doi.org/10.1109/SPIN.2018.8474179Google ScholarCross Ref
- John McCaffery, Alan Miller, Anna Vermehren, and Adeola Fabola. 2015. The Virtual Museums of Caen: A case study on modes of representation of digital historical content. In 2015 Digital Heritage, Vol. 2. 541–548. https://doi.org/10.1109/DigitalHeritage.2015.7419571Google ScholarCross Ref
- Danial Moazen, Seyed A Sajjadi, and Ani Nahapetian. 2016. AirDraw: Leveraging smart watch motion sensors for mobile human computer interactions. In 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC). 442–446. https://doi.org/10.1109/CCNC.2016.7444820Google ScholarDigital Library
- Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815Google ScholarDigital Library
- T. H. Nascimento, C. B. R. Ferreira, Wellington G. Rodrigues, and Fabrizzio Soares. 2020. Interaction with Smartwatches Using Gesture Recognition: A Systematic Literature Review. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). 1661–1666. https://doi.org/10.1109/COMPSAC48688.2020.00-17Google ScholarCross Ref
- Thamer Horbylon Nascimento, Fabrizzio Soares, Hugo Nascimento, Rogerio Salvini, Mateus Luna, Cristhiane Gonçalves, and Eduardo Souza. 2018. Interaction with Platform Games Using Smartwatches and Continuous Gesture Recognition: A Case Study. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 02. 253–258. https://doi.org/10.1109/COMPSAC.2018.10239Google ScholarCross Ref
- Thamer Horbylon Nascimento, Fabrizzio Soares, Hugo Nascimento, Marcos Vieira, Thyago Carvalho, and Wesley Miranda. 2019. Netflix Control Method Using Smartwatches and Continuous Gesture Recognition. In 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE). 1–4. https://doi.org/10.1109/CCECE.2019.8861610Google ScholarCross Ref
- Thamer Horbylon Nascimento, Fabrizzio Soares, Danilo Vieira Oliveira, Rogerio Lopes Salvini, Ronaldo Martins da Costa, and Cristhiane Gonçalves. 2017. Method for Text Input with Google Cardboard: An Approach Using Smartwatches and Continuous Gesture Recognition. In 2017 19th Symposium on Virtual and Augmented Reality (SVR). 223–226. https://doi.org/10.1109/SVR.2017.36Google ScholarCross Ref
- Sang-Min Park and Young-Gab Kim. 2022. A Metaverse: Taxonomy, Components, Applications, and Open Challenges. IEEE Access 10 (2022), 4209–4251. https://doi.org/10.1109/ACCESS.2021.3140175Google ScholarCross Ref
- I. Picallo, A. Vidal-Balea, O. Blanco-Novoa, P. Lopez-Iturri, P. Fraga-Lamas, H. Klaina, T. M. Fernández-Caramés, L. Azpilicueta, and F. Falcone. 2021. Design and Experimental Validation of an Augmented Reality System With Wireless Integration for Context Aware Enhanced Show Experience in Auditoriums. IEEE Access 9 (2021), 5466–5484. https://doi.org/10.1109/ACCESS.2020.3048203Google ScholarCross Ref
- Xiaogang Ruan and Chongyang Tian. 2015. Dynamic gesture recognition based on improved DTW algorithm. In 2015 IEEE International Conference on Mechatronics and Automation (ICMA). 2134–2138. https://doi.org/10.1109/ICMA.2015.7237816Google ScholarCross Ref
- Namrata Singh and Sarvpal Singh. 2017. Virtual reality: A brief survey. In 2017 International Conference on Information Communication and Embedded Systems (ICICES). 1–6. https://doi.org/10.1109/ICICES.2017.8070720Google ScholarCross Ref
- Gineke A Ten Holt, Marcel JT Reinders, and Emile A Hendriks. 2007. Multi-dimensional dynamic time warping for gesture recognition. In Thirteenth annual conference of the Advanced School for Computing and Imaging, Vol. 300. 1.Google Scholar
- Pui Chung Wong, Kening Zhu, Xing-Dong Yang, and Hongbo Fu. 2020. Exploring Eyes-Free Bezel-Initiated Swipe on Round Smartwatches. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376393Google ScholarDigital Library
- Wenge Xu, Hai-Ning Liang, Yuxuan Zhao, Tianyu Zhang, Difeng Yu, and Diego Monteiro. 2019. RingText: Dwell-free and hands-free Text Entry for Mobile Head-Mounted Displays using Head Motions. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1991–2001. https://doi.org/10.1109/TVCG.2019.2898736Google ScholarCross Ref
- Huixin Zhang, Ya Hu, Jun Zhu, Lin Fu, Bingli Xu, and Weilian Li. 2022. A gaze-based interaction method for large-scale and large-space disaster scenes within mobile virtual reality. Transactions in GIS 26, 3 (2022).Google ScholarCross Ref
- Yuzhou Zhuang, Yuntao Wang, Yukang Yan, Xuhai Xu, and Yuanchun Shi. 2021. ReflecTrack: Enabling 3D Acoustic Position Tracking Using Commodity Dual-Microphone Smartphones. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 1050–1062. https://doi.org/10.1145/3472749.3474805Google ScholarDigital Library
Index Terms
- Exploring Interaction in a Virtual Music Studio through Gesture Recognition on Smartwatches and HMD Devices
Recommendations
Integration of Smartwatches in Low-Cost Virtual Environments: A Systematic Literature Review with Emphasis on Cardboard Devices
SVR '23: Proceedings of the 25th Symposium on Virtual and Augmented RealityIn this work, we investigated the integration between smartwatches and Cardboard devices for interaction in low-cost virtual environments. We analyzed five related works that explored this integration and discussed the main trends and challenges ...
DrumsVR: Simulating Drum Percussion in a Virtual Environment Using Gesture Recognition on Smartwatches
Web3D '23: Proceedings of the 28th International ACM Conference on 3D Web TechnologyThis work presents the development of DrumsVR, a system for simulating drum percussion using gesture recognition on smartwatches and virtual reality. The objective is to provide an immersive and interactive experience for users, allowing them to play a ...
MazeVR: Immersion and Interaction Using Google Cardboard and Continuous Gesture Recognition on Smartwatches
Web3D '23: Proceedings of the 28th International ACM Conference on 3D Web TechnologyThis work presents the development and usability evaluation of the MazeVR game, a maze game designed for the Google Cardboard device. The game allows interactions through head movements to control the camera and gestures on the smartwatch to perform ...
Comments