Mixed Reality Approach to Serious Games in Developing Musical Intelligence: A Systematic Mapping Study
Resumo
This study presents a systematic review investigating the effectiveness of Mixed Reality (MR)-based serious games in fostering musical intelligence, particularly in the domains of rhythm and melody recognition. By applying the SEGRESS methodology, the authors reviewed 466 articles, ultimately selecting 23 primary studies that met stringent inclusion criteria. The review synthesizes findings across several dimensions, including targeted musical skills, user demographics, technological platforms, assessment methods, and measurable outcomes. Results indicate that most MR serious games emphasize receptive and reproductive musical skills, particularly pitch and melody recognition, with rhythm training comparatively underrepresented. Target audiences predominantly consist of beginners and children, and mobile-based Augmented Reality (AR) emerges as a preferred platform for accessibility. Although studies employed diverse evaluation techniques—ranging from pre/post-tests to user feedback—the majority reported positive impacts on musical learning and engagement. However, gaps persist in research involving intermediate and advanced learners, standardized assessments of musical intelligence, and long-term pedagogical outcomes. This review contributes a foundational synthesis of current evidence and outlines future directions for enhancing the integration of immersive technology in music education.
Palavras-chave:
mixed reality, augmented reality, serious games, music education, musical intelligence, rhythm recognition, melody recognition, educational technology
Referências
Xin Wang, Pramote Phokha, and Nut Chiangthong. A comparative study on music curriculum design in universities based on ar technology: A survey of 10 undergraduate colleges in nanjing. Journal of Ecohumanism, 3(7):3962–3975, Oct. 2024.
Yaqi Zhang and Yao Song. The effects of sensory cues on immersive experiences for fostering technology-assisted sustainable behavior: A systematic review. Behavioral Sciences, 12(10), 2022.
C.C. Abt. Serious Games. Viking compass book. Viking Press, 1970.
Damien Djaouti, Julian Alvarez, Jean-Pierre Jessel, and Olivier Rampnoux. Origins of Serious Games, pages 25–43. Springer London, London, 2011.
H. Gardner. Frames of Mind: The Theory of Multiple Intelligences. Basic Books, 2011.
Valéria Farinazzo Martins, Letícia Gomez, and Ana Grasielle Dionísio Corrêa. Teaching children musical perception with MUSIC-AR. ICST Trans. e-Educ. e-Learn., 2(5):e3, March 2015.
Imane Salhi, Martyna Poreba, Erwan Piriou, Valerie Gouet-Brunet, and Maroun Ojail. Chapter 8 - multimodal localization for embedded systems: A survey. In Michael Ying Yang, Bodo Rosenhahn, and Vittorio Murino, editors, Multimodal Scene Understanding, pages 199–278. Academic Press, 2019.
Fumio KISHINO Paul MILGRAM. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information, E77-D(12):1321–1329, December 1994.
Vijaykumar Krishnan, Karen Machleit, James Kellaris, Ursula Sullivan, and Timothy Aurand. Musical intelligence: Explication, measurement, and implications for consumer behavior. Journal of Consumer Marketing, 31, 06 2014.
Lap-Kei Lee, Xiaodong Wei, Kwok Tai Chui, Simon K. S. Cheung, Fu Lee Wang, Yin-Chun Fung, Angel Lu, Yan Keung Hui, Tianyong Hao, Leong Hou U, and Nga-In Wu. A systematic review of the design of serious games for innovative learning: Augmented reality, virtual reality, or mixed reality? Electronics, 13(5), 2024.
Barbara Kitchenham, Lech Madeyski, and David Budgen. Segress: Software engineering guidelines for reporting secondary studies. IEEE Transactions on Software Engineering, 49(3):1273–1298, 2023.
Roberto Rufino Júnior, Tadeu Moreira de Classe, Rodrigo Pereira dos Santos, and Sean Wolfgand Matsui Siqueira. Current risk situation training in industry, and games as a strategy for playful, engaging and motivating training. Journal on Interactive Systems, 14(1):138–156, 2023.
Sebastian Rutkowski, Mateusz Adamczyk, Agnieszka Pastuła, Edyta Gos, Carlos Luque-Moreno, and Anna Rutkowska. Training using a commercial immersive virtual reality system on hand–eye coordination and reaction time in young musicians: A pilot study. International Journal of Environmental Research and Public Health, 18(3), 2021.
Matevz Pesek, Nejc Hirci, Klara Znidersic, and Matija Marolt. Enhancing music rhythmic perception and performance with a vr game. VIRTUAL REALITY, 28(2), MAY 25 2024.
Sebastian Rutkowski, Aleksandra Jakóbczyk, Kacper Abrahamek, Aleksandra Nowakowska, Magdalena Nowak, David Liska, Ladislav Batalik, Vera Colombo, and Marco Sacco. Training using a commercial immersive virtual reality system on hand–eye coordination and reaction time in students: a randomized controlled trial. Virtual Real., 28(1), March 2024.
Edilson M Chaves, Paulo Bruno de A Braga, Yuri Fontenelle L Montenegro, Vitória B Rodrigues, Marilene C Munguba, and Victor Hugo C Albuquerque. Toc-Tum mini-games: An educational game accessible for deaf culture based on virtual reality. Expert Syst., 38(1), January 2021.
Simon Senecal, Niels A Nijdam, Andreas Aristidou, and Nadia Magnenat-Thalmann. Salsa dance learning evaluation and motion analysis in gamified virtual reality environment. Multimed. Tools Appl., 79(33-34):24621–24643, September 2020.
Gavin Gosling, Ivan-Teofil Catovic, Ghazal Bangash, Daniel MacCormick, and Loutfouz Zaman. A comparison of interfaces for learning how to play a mixed reality handpan. In 2023 IEEE Gaming, Entertainment, and Media Conference (GEM), pages 1–6. IEEE, November 2023.
Federico Avanzini, Adriano Baratè, Mauro Cottini, Luca Andrea Ludovico, and Marcella Mandanici. Developing music harmony awareness in young students through an augmented reality approach. In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications. SCITEPRESS - Science and Technology Publications, 2020.
Jie-Syu Yang and Chien-Wen Cheng. A pitch perception training game in VR environment for enhancing music learning motivation. In 2020 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan). IEEE, September 2020.
Yi Lu, Xiaoye Wang, Jiangtao Gong, and Yun Liang. ChordAR: An educational AR game design for children’s music theory learning. Wirel. Commun. Mob. Comput., 2022:1–9, February 2022.
Ana Grasielle Dionisio Correa, Gilda Aparecida de Assis, Marilena do Nascimento, Irene Ficheman, and Roseli de Deus Lopes. Genvirtual:: An augmented reality musical game for cognitive and motor rehabilitation. In 2007 VIRTUAL REHABILITATION, pages 1+, 2007. Virtual Rehabilitation Conference 2007, Venice, ITALY, SEP 27-29, 2007.
Leonard Bruns, Benedict Saurbier, Tray Minh Voong, and Michael Oehler. Presence and flow in virtual and mixed realities for music-related educational settings. In 2024 IEEE 5TH INTERNATIONAL SYMPOSIUM ON THE INTERNET OF SOUNDS, IS2 2024, pages 280–286. IEEE; IEEE Commun Soc; Genelec; Samsung; Sennheiser; Qualcomm, 2024. 5th IEEE International Symposium on the Internet of Sounds (IEEE IS2), Int Audio Labs Erlangen, Erlangen, GERMANY, SEP 30-OCT 02, 2024.
Myunjin Park, Kyujung Kim, and Dongwuk Kyoung. Prospective design of interactive emotional sound edutainment system: Mesig. In MCCSIS’08 - IADIS Multi Conference on Computer Science and Information Systems; Proceedings of e-Learning 2008, volume 1, page 83 – 90, 2008. Cited by: 0.
Myunjin Park and Kyujung Kim. Design of interactive emotional sound edutainment system. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5613 LNCS(PART 4):368 – 377, 2009. Cited by: 0.
Jonathan Chow, Haoyang Feng, Robert Amor, and Burkhard C. Wünsche. Music education using augmented reality with a head mounted display. In Conferences in Research and Practice in Information Technology Series, volume 139, page 73 – 80, 2013. Cited by: 54.
Garyfallia Preka and Maria Rangoussi. Augmented reality and qr codes for teaching music to preschoolers and kindergarteners: Educational intervention and evaluation. In Lane H., Zvacek S., and Uhomoibhi J., editors, CSEDU 2019 - Proceedings of the 11th International Conference on Computer Supported Education, volume 1, page 113 – 123. SciTePress, 2019. Cited by: 13; All Open Access, Green Open Access, Hybrid Gold Open Access.
Rubén Jesús García Hernández, Isabel Barbancho Pérez, Lorenzo José Tardón García, Jon Arambarri, Milán Magdics, and Mateu Sbert. E-cecilia: Implementation of a music game. In Camacho D., Gonzalez Calero P.A., and Gomez Martin M.A., editors, CEUR Workshop Proceedings, volume 1196, page 66 – 77. CEUR-WS, 2014. Cited by: 4.
Joyce H. Fonteles, Maria Stephanie S. Fernandes, Daniel L. Queiroz, and Maria Andreia F. Rodrigues. Interactive musical game with a gesture-controlled virtual puppet. In Nunes F.L.S. and de Oliveira J.C., editors, Proceedings - 19th Symposium on Virtual and Augmented Reality, SVR 2017, volume 2017-November, page 261 – 270. Institute of Electrical and Electronics Engineers Inc., 2017. Cited by: 3.
Svitlana Antoshchuk, Mykyta Kovalenko, and Jürgen Sieck. Creating an interactive musical experience for a concert hall. International Journal of Computing, 17(3):143 – 152, 2018. Cited by: 4; All Open Access, Hybrid Gold Open Access.
Shantanu Das, Seth Glickman, Fu Yen Hsiao, and Byunghwan Lee. Music everywhere – augmented reality piano improvisation learning system. In Proceedings of the International Conference on New Interfaces for Musical Expression, page 511 – 512. International Conference on New Interfaces for Musical Expression, 2017. Cited by: 20.
Letícia Gomes, Valéria Farinazzo Martins, Diego Colombo Dias, and Marcelo De Paiva Guimarães. Music-ar: Augmented reality in teaching the concept of sound loudness to children in pre-school; [music-ar: Realidade aumentada no ensino do conceito de altura sonora à crianças da pré-escola]. In Proceedings - 2014 16th Symposium on Virtual and Augmented Reality, SVR 2014, page 114 – 117. Institute of Electrical and Electronics Engineers Inc., 2014. Cited by: 16.
Ismail Mohamed Jamal and Erdal Kilic. Easyarpiano: Piano teaching mobile app with augmented reality. In Proceedings - 2021 International Conference on Forthcoming Networks and Sustainability in AIoT Era, FoNeS-AIoT 2021, page 66 – 71. Institute of Electrical and Electronics Engineers Inc., 2021. Cited by: 5.
Abdul Syukur, Pulung Nurtantio Andono, and Arry Maulana Syarif. A multiplayer virtual reality game for playing gamelan music in a virtual orchestra mode. In 2023 3rd International Conference on Intelligent Cybernetics Technology and Applications, ICICyTA 2023, page 375 – 379. Institute of Electrical and Electronics Engineers Inc., 2023. Cited by: 1.
Yaqi Zhang and Yao Song. The effects of sensory cues on immersive experiences for fostering technology-assisted sustainable behavior: A systematic review. Behavioral Sciences, 12(10), 2022.
C.C. Abt. Serious Games. Viking compass book. Viking Press, 1970.
Damien Djaouti, Julian Alvarez, Jean-Pierre Jessel, and Olivier Rampnoux. Origins of Serious Games, pages 25–43. Springer London, London, 2011.
H. Gardner. Frames of Mind: The Theory of Multiple Intelligences. Basic Books, 2011.
Valéria Farinazzo Martins, Letícia Gomez, and Ana Grasielle Dionísio Corrêa. Teaching children musical perception with MUSIC-AR. ICST Trans. e-Educ. e-Learn., 2(5):e3, March 2015.
Imane Salhi, Martyna Poreba, Erwan Piriou, Valerie Gouet-Brunet, and Maroun Ojail. Chapter 8 - multimodal localization for embedded systems: A survey. In Michael Ying Yang, Bodo Rosenhahn, and Vittorio Murino, editors, Multimodal Scene Understanding, pages 199–278. Academic Press, 2019.
Fumio KISHINO Paul MILGRAM. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information, E77-D(12):1321–1329, December 1994.
Vijaykumar Krishnan, Karen Machleit, James Kellaris, Ursula Sullivan, and Timothy Aurand. Musical intelligence: Explication, measurement, and implications for consumer behavior. Journal of Consumer Marketing, 31, 06 2014.
Lap-Kei Lee, Xiaodong Wei, Kwok Tai Chui, Simon K. S. Cheung, Fu Lee Wang, Yin-Chun Fung, Angel Lu, Yan Keung Hui, Tianyong Hao, Leong Hou U, and Nga-In Wu. A systematic review of the design of serious games for innovative learning: Augmented reality, virtual reality, or mixed reality? Electronics, 13(5), 2024.
Barbara Kitchenham, Lech Madeyski, and David Budgen. Segress: Software engineering guidelines for reporting secondary studies. IEEE Transactions on Software Engineering, 49(3):1273–1298, 2023.
Roberto Rufino Júnior, Tadeu Moreira de Classe, Rodrigo Pereira dos Santos, and Sean Wolfgand Matsui Siqueira. Current risk situation training in industry, and games as a strategy for playful, engaging and motivating training. Journal on Interactive Systems, 14(1):138–156, 2023.
Sebastian Rutkowski, Mateusz Adamczyk, Agnieszka Pastuła, Edyta Gos, Carlos Luque-Moreno, and Anna Rutkowska. Training using a commercial immersive virtual reality system on hand–eye coordination and reaction time in young musicians: A pilot study. International Journal of Environmental Research and Public Health, 18(3), 2021.
Matevz Pesek, Nejc Hirci, Klara Znidersic, and Matija Marolt. Enhancing music rhythmic perception and performance with a vr game. VIRTUAL REALITY, 28(2), MAY 25 2024.
Sebastian Rutkowski, Aleksandra Jakóbczyk, Kacper Abrahamek, Aleksandra Nowakowska, Magdalena Nowak, David Liska, Ladislav Batalik, Vera Colombo, and Marco Sacco. Training using a commercial immersive virtual reality system on hand–eye coordination and reaction time in students: a randomized controlled trial. Virtual Real., 28(1), March 2024.
Edilson M Chaves, Paulo Bruno de A Braga, Yuri Fontenelle L Montenegro, Vitória B Rodrigues, Marilene C Munguba, and Victor Hugo C Albuquerque. Toc-Tum mini-games: An educational game accessible for deaf culture based on virtual reality. Expert Syst., 38(1), January 2021.
Simon Senecal, Niels A Nijdam, Andreas Aristidou, and Nadia Magnenat-Thalmann. Salsa dance learning evaluation and motion analysis in gamified virtual reality environment. Multimed. Tools Appl., 79(33-34):24621–24643, September 2020.
Gavin Gosling, Ivan-Teofil Catovic, Ghazal Bangash, Daniel MacCormick, and Loutfouz Zaman. A comparison of interfaces for learning how to play a mixed reality handpan. In 2023 IEEE Gaming, Entertainment, and Media Conference (GEM), pages 1–6. IEEE, November 2023.
Federico Avanzini, Adriano Baratè, Mauro Cottini, Luca Andrea Ludovico, and Marcella Mandanici. Developing music harmony awareness in young students through an augmented reality approach. In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications. SCITEPRESS - Science and Technology Publications, 2020.
Jie-Syu Yang and Chien-Wen Cheng. A pitch perception training game in VR environment for enhancing music learning motivation. In 2020 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan). IEEE, September 2020.
Yi Lu, Xiaoye Wang, Jiangtao Gong, and Yun Liang. ChordAR: An educational AR game design for children’s music theory learning. Wirel. Commun. Mob. Comput., 2022:1–9, February 2022.
Ana Grasielle Dionisio Correa, Gilda Aparecida de Assis, Marilena do Nascimento, Irene Ficheman, and Roseli de Deus Lopes. Genvirtual:: An augmented reality musical game for cognitive and motor rehabilitation. In 2007 VIRTUAL REHABILITATION, pages 1+, 2007. Virtual Rehabilitation Conference 2007, Venice, ITALY, SEP 27-29, 2007.
Leonard Bruns, Benedict Saurbier, Tray Minh Voong, and Michael Oehler. Presence and flow in virtual and mixed realities for music-related educational settings. In 2024 IEEE 5TH INTERNATIONAL SYMPOSIUM ON THE INTERNET OF SOUNDS, IS2 2024, pages 280–286. IEEE; IEEE Commun Soc; Genelec; Samsung; Sennheiser; Qualcomm, 2024. 5th IEEE International Symposium on the Internet of Sounds (IEEE IS2), Int Audio Labs Erlangen, Erlangen, GERMANY, SEP 30-OCT 02, 2024.
Myunjin Park, Kyujung Kim, and Dongwuk Kyoung. Prospective design of interactive emotional sound edutainment system: Mesig. In MCCSIS’08 - IADIS Multi Conference on Computer Science and Information Systems; Proceedings of e-Learning 2008, volume 1, page 83 – 90, 2008. Cited by: 0.
Myunjin Park and Kyujung Kim. Design of interactive emotional sound edutainment system. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5613 LNCS(PART 4):368 – 377, 2009. Cited by: 0.
Jonathan Chow, Haoyang Feng, Robert Amor, and Burkhard C. Wünsche. Music education using augmented reality with a head mounted display. In Conferences in Research and Practice in Information Technology Series, volume 139, page 73 – 80, 2013. Cited by: 54.
Garyfallia Preka and Maria Rangoussi. Augmented reality and qr codes for teaching music to preschoolers and kindergarteners: Educational intervention and evaluation. In Lane H., Zvacek S., and Uhomoibhi J., editors, CSEDU 2019 - Proceedings of the 11th International Conference on Computer Supported Education, volume 1, page 113 – 123. SciTePress, 2019. Cited by: 13; All Open Access, Green Open Access, Hybrid Gold Open Access.
Rubén Jesús García Hernández, Isabel Barbancho Pérez, Lorenzo José Tardón García, Jon Arambarri, Milán Magdics, and Mateu Sbert. E-cecilia: Implementation of a music game. In Camacho D., Gonzalez Calero P.A., and Gomez Martin M.A., editors, CEUR Workshop Proceedings, volume 1196, page 66 – 77. CEUR-WS, 2014. Cited by: 4.
Joyce H. Fonteles, Maria Stephanie S. Fernandes, Daniel L. Queiroz, and Maria Andreia F. Rodrigues. Interactive musical game with a gesture-controlled virtual puppet. In Nunes F.L.S. and de Oliveira J.C., editors, Proceedings - 19th Symposium on Virtual and Augmented Reality, SVR 2017, volume 2017-November, page 261 – 270. Institute of Electrical and Electronics Engineers Inc., 2017. Cited by: 3.
Svitlana Antoshchuk, Mykyta Kovalenko, and Jürgen Sieck. Creating an interactive musical experience for a concert hall. International Journal of Computing, 17(3):143 – 152, 2018. Cited by: 4; All Open Access, Hybrid Gold Open Access.
Shantanu Das, Seth Glickman, Fu Yen Hsiao, and Byunghwan Lee. Music everywhere – augmented reality piano improvisation learning system. In Proceedings of the International Conference on New Interfaces for Musical Expression, page 511 – 512. International Conference on New Interfaces for Musical Expression, 2017. Cited by: 20.
Letícia Gomes, Valéria Farinazzo Martins, Diego Colombo Dias, and Marcelo De Paiva Guimarães. Music-ar: Augmented reality in teaching the concept of sound loudness to children in pre-school; [music-ar: Realidade aumentada no ensino do conceito de altura sonora à crianças da pré-escola]. In Proceedings - 2014 16th Symposium on Virtual and Augmented Reality, SVR 2014, page 114 – 117. Institute of Electrical and Electronics Engineers Inc., 2014. Cited by: 16.
Ismail Mohamed Jamal and Erdal Kilic. Easyarpiano: Piano teaching mobile app with augmented reality. In Proceedings - 2021 International Conference on Forthcoming Networks and Sustainability in AIoT Era, FoNeS-AIoT 2021, page 66 – 71. Institute of Electrical and Electronics Engineers Inc., 2021. Cited by: 5.
Abdul Syukur, Pulung Nurtantio Andono, and Arry Maulana Syarif. A multiplayer virtual reality game for playing gamelan music in a virtual orchestra mode. In 2023 3rd International Conference on Intelligent Cybernetics Technology and Applications, ICICyTA 2023, page 375 – 379. Institute of Electrical and Electronics Engineers Inc., 2023. Cited by: 1.
Publicado
15/09/2025
Como Citar
FONTANA, Paulo Gabriel Alves; CLASSE, Tadeu Moreira de.
Mixed Reality Approach to Serious Games in Developing Musical Intelligence: A Systematic Mapping Study. In: SIMPÓSIO BRASILEIRO DE COMPUTAÇÃO MUSICAL (SBCM), 19. , 2025, Campinas/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 120-127.
DOI: https://doi.org/10.5753/sbcm.2025.11912.
