HeroKeys: a gamified virtual environment for playing and learning piano

• Deivison R. Jordão UFPB
• Joaquim B. Cavalcante UFPB
• Humberto Navarro de Carvalho Music.AI
• Yvson Nunes Music.AI
• Thais Gaudencio ARIA / UFPB
• Yuri Malheiros ARIA / UFPB

DOI: https://doi.org/10.5753/webmedia_estendido.2025.16344

Abstract

Musical education remains inaccessible to many due to the reliance on traditional methods, which demand sheet music, prior knowledge, or formal training. This paper presents HeroKeys, an open-source gamified web application designed to enhance piano learning through artificial intelligence-based audio transcription. Unlike traditional learning methods that rely on pre-existing transcriptions, HeroKeys allows users to practice any song of their choice by automatically, converting audio files into playable musical notation. The system features multiple visualization options including Piano Roll, traditional sheet music, and a display of the current chord being played, accommodating different learning preferences. The application incorporates gamification elements such as scoring systems, multipliers, and visual feedback to increase user engagement and motivation. HeroKeys supports external MIDI keyboard connections and provides customizable playback controls to facilitate progressive learning. Our approach combines audio processing technologies with interactive elements to create an accessible and motivating piano learning environment, making musical education more engaging and accessible to beginners.

Keywords: Aprendizagem de piano, transcrição musical, inteligência artificial, gamificação

References

Marli Appel-Silva, Guilherme Welter Wendt, and Irani Iracema de Lima Argimon. 2010. A teoria da autodeterminação e as influências socioculturais sobre a identidade. Psicol. rev. (Belo Horizonte) 16, 2 (ago 2010), 351–369. [link] Acesso em: 27 set. 2025.

Estefan Cano, Derry FitzGerald, Antoine Liutkus, Mark D. Plumbley, and Fabian Stöter. 2019. Musical source separation: An introduction. IEEE Signal Processing Magazine 36, 1 (2019), 31–40. DOI: 10.1109/MSP.2018.2874719

Mihaly Csikszentmihalyi. 2014. Teoria do Flow, pesquisa e aplicações. ComCiência 161 (set 2014). [link] Acesso em: 27 set. 2025.

FastAPI. 2025. FastAPI framework, high performance, easy to learn, fast to code, ready for production. [link]. Acesso em: jul. 2025.

Harmonix Music Systems. 2005. Guitar Hero. Video Game. PlayStation 2.

klang.io. 2006. Transcribe Piano Music into Notes. Retrieved July 7, 2025 from [link]

Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. 2020. High-resolution Piano Transcription with Pedals by Regressing Onsets and Offsets Times. CoRR abs/2010.01815 (2020). arXiv:2010.01815 [link]

W. Molloy, E. Huang, and B. C. Wünsche. 2019. Mixed Reality Piano Tutor: A Gamified Piano Practice Environment. In Proceedings of the International Conference on Electronics, Information, and Communication (ICEIC). Auckland, New Zealand, 1–7. DOI: 10.23919/ELINFOCOM.2019.8706474

Music.AI. 2025. AI Audio Models to Power Your Music Business. [link]. Acesso em: jul. 2025.

Next.js. 2025. The React Framework for the Web. [link]. Acesso em: jul. 2025.

I. G. Putranda, A. M. Yumna, Y. Rosmansyah, and Y. Sukmana. 2023. Exploring Audio Processing in Mixed Reality to Boost Motivation in Piano Learning. IEEE Access 11 (2023), 71194–71200. DOI: 10.1109/ACCESS.2023.3293250

Fabiola Rosendo Vignola, Nancy Domínguez González, Daniel Serna Poot, and Ernesto Vilches Lleó. 2023. Estrategia de gamificación para la enseñanza de solfeo a niños. RIDE. Rev. Iberoam. Investig. Desarro. Educ 14, 27 (dez 2023), e536. DOI: 10.23913/ride.v14i27.1611 Epub 22-Jan-2024. Acesso em: 27 set. 2025.