Thunder: A Design Process to Build Emotionally Engaging Music Visualizations
Abstract
Music's profound emotional impact extends beyond traditional listening experiences, playing a critical role in shaping user engagement in entertainment contexts, including digital games. Music visualizations -- graphical representations of musical elements such as rhythm, melody, and harmony -- are becoming increasingly significant for enriching emotional depth in these contexts. Despite the potential of music to enhance digital media experiences, there remains a gap in the intentional integration of musical visualizations with specific emotional narratives. In this paper, we propose Thunder, a design process for creating musical visualizations that prioritize conveying the emotional communication of music. Our approach builds on an existing method for rapid prototyping of animated musical visualizations and comprises four iterative phases. By applying this process in creating two music visualizations, we demonstrate that our process aligns closely with the designed emotional content of the music. The results showed that our alternative visualizations effectively communicated intended negative emotions, such as "sadness" and "shame", and were well-received by the public, highlighting the importance of synchronized visual and musical elements. With this work, we hope to offer valuable insights for game designers and entertainment app developers, providing a comprehensive design process for crafting musical visualization experiences that emphasize the emotional richness of music and pave the way for more immersive and emotionally resonant digital media experiences.
Keywords:
Animated Music Visualization, Design Process, Emotions, Evaluation
References
1940. Fantasia.
1976. Atari Video Music
2018. Design Thinking. [link]
2019. Beat Saber. [link]
Horhanna Almeida, Giordano Cabral, and Rute Moura. 2021. Design process and rapid prototyping of animated music visualizations. In Anais do XVIII Simpósio Brasileiro de Computação Musical. SBC, 114–120.
Robert J Brown. 1978. Audio activated video display. US Patent 4,081,829.
Gabriel Dias Cantareira, Luis Gustavo Nonato, and Fernando V. Paulovich. 2016. MoshViz: A Detail+Overview Approach to Visualize Music Elements. IEEE Transactions on Multimedia 18, 11 (2016), 2238–2246. DOI: 10.1109/TMM.2016.2614226
Mona Lisa Chanda and Daniel J Levitin. 2013. The neurochemistry of music. Trends in cognitive sciences 17, 4 (2013), 179–193.
Marshall Chasin. 2003. Music and hearing aids. The Hearing Journal 56, 7 (2003), 36–38.
Chuer Chen, Nan Cao, Jiani Hou, Yi Guo, Yulei Zhang, and Yang Shi. 2023. MusicJam: Visualizing Music Insights via Generated Narrative Illustrations. arXiv preprint arXiv:2308.11329 (2023).
Alan S Cowen, Xia Fang, Disa Sauter, and Dacher Keltner. 2020. What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures. Proceedings of the National Academy of Sciences 117, 4 (2020), 1924–1934.
Caroline Cuny, M. Fornerino, and A. Helme-Guizon. 2015. Can music improve e-behavioral intentions by enhancing consumers’ immersion and experience? Inf. Manag. 52 (2015), 1025–1034. DOI: 10.1016/j.im.2015.07.009
Roberto De Prisco, Delfina Malandrino, Donato Pirozzi, Gianluca Zaccagnino, and Rocco Zaccagnino. 2018. Evaluation study of visualisations for harmonic analysis of 4-part music. In 2018 22nd International Conference Information Visualisation (IV). IEEE, 484–489.
Pieter Desmet. 2018. Measuring emotion: Development and application of an instrument to measure emotional responses to products. Funology 2: From Usability to Enjoyment (2018), 391–404.
Ed Diener and Micaela Y Chan. 2011. Happy people live longer: Subjective well-being contributes to health and longevity. Applied Psychology: Health and Well-Being 3, 1 (2011), 1–43
David W Fourney and Deborah I Fels. 2009. Creating access to music through visualization. In 2009 ieee toronto international conference science and technology for humanity (tic-sth). IEEE, 939–944.
Susan Hallam and Raymond MacDonald. 2013. Introduction: Perspectives on the power of music. , 83–86 pages.
Marc Hassenzahl. 2008. User experience (UX): towards an experiential perspective on product quality. (2008), 11–15. DOI: 10.1145/1512714.1512717
Rumi Hiraga, Fumiko Watanabe, and Issei Fujishiro. 2002. Music learning through visualization. In Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings. IEEE, 101–108.
Jukka Holm and Harri Siirtola. 2012. A Comparison of Methods for Visualizing Musical Genres. In 2012 16th International Conference on Information Visualisation. 636–645. DOI: 10.1109/IV.2012.107
Felicia A Huppert. 2009. Psychological well-being: Evidence regarding its causes and consequences. Applied psychology: health and well-being 1, 2 (2009), 137–164
A. Hwang and Jeeyun Oh. 2020. Interacting with background music engages E-Customers more: The impact of interactive music on consumer perception and behavioral intention. Journal of Retailing and Consumer Services 54 (2020), 101928. DOI: 10.1016/j.jretconser.2019.101928
Patrik N Juslin and Petri Laukka. 2004. Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of new music research 33, 3 (2004), 217–238.
Patrik N Juslin and Daniel Västfjäll. 2008. Emotional responses to music: The need to consider underlying mechanisms. Behavioral and brain sciences 31, 5 (2008), 559–575.
Hiroharu Kamioka, Kiichiro Tsutani, Minoru Yamada, Hyuntae Park, Hiroyasu Okuizumi, Koki Tsuruoka, Takuya Honda, Shinpei Okada, Sang-Jun Park, Jun Kitayuguchi, et al. 2014. Effectiveness of music therapy: a summary of systematic reviews based on randomized controlled trials of music interventions. Patient preference and adherence (2014), 727–754.
Corey LM Keyes. 2007. Promoting and protecting mental health as flourishing: a complementary strategy for improving national mental health. American psychologist 62, 2 (2007), 95.
Stefan Koelsch. 2010. Towards a neural basis of music-evoked emotions. Trends in cognitive sciences 14, 3 (2010), 131–137.
Stefan Koelsch. 2014. Brain Correlate of music-evoked emotions. Nature reviews. Neuroscience 15 (02 2014), 170–180. DOI: 10.1038/nrn3666
Stefan Koelsch. 2015. Music-evoked emotions: principles, brain correlates, and implications for therapy. Annals of the New York Academy of Sciences 1337, 1 (2015), 193–201.
Yunli Lee and Revina Nur Fathia. 2016. Interactive music visualization for music player using processing. In 2016 22nd International Conference on Virtual System & Multimedia (VSMM). IEEE, 1–4.
Hugo B Lima, Carlos GR Dos Santos, and Bianchi S Meiguins. 2021. A survey of music visualization techniques. ACM Computing Surveys (CSUR) 54, 7 (2021), 1–29.
Scott D Lipscomb and Roger A Kendall. 1994. Perceptual judgement of the relationship between musical and visual components in film. Psychomusicology: A Journal of Research in Music Cognition 13, 1-2 (1994), 60.
M. Matessino. 1999. Letter in response to ’A Study of Jaws’ incisive overture to close off the century’. Internet. [link] [Accessed on March 4, 2009].
David Miranda, Carmen Li, and Ticianne Darin. 2021. UES-Br: Translation and Cross-Cultural Adaptation of the User Engagement Scale for Brazilian Portuguese. Proceedings of the ACM on Human-Computer Interaction 5 (10 2021), 1–22. 10.1145/3474705
Luminita Heliana Munteanu. 2012. Musical culture, a finality of musical education. Procedia-Social and Behavioral Sciences 46 (2012), 4195–4199.
D. Pereira and T. Chambel. 2023. Enhancing Emotional Awareness and Regulation in Movies and Music Based on Personality. Proceedings of the 2023 ACM International Conference on Interactive Media Experiences (2023). DOI: 10.1145/3573381.3596462
Caio Pereira Nunes and Ticianne Darin. 2024. Cross-cultural adaptation of the Intrinsic Motivation Inventory Task Evaluation Questionnaire into Brazilian Portuguese. 1–11. DOI: 10.1145/3638067.3638083
Ramesh Pokharel. 2020. Exploring the Different Roles and Aspects of Music. Molung Educational Frontier (2020), 125–133.
Michael Pouris and Deborah I Fels. 2012. Creating an entertaining and informative music visualization. In International Conference on Computers for Handicapped Persons. Springer, 451–458.
Jennifer Preece, Yvonne Rogers, and Helen Sharp. 2013. Design de Interação: além da interação humano-computador. Bookman.
Roberto De Prisco, Delfina Malandrino, Donato Pirozzi, Gianluca Zaccagnino, and Rocco Zaccagnino. 2017. Understanding the structure of musical compositions: Is visualization an effective approach? Information Visualization 16, 2 (2017), 139–152. DOI: 10.1177/1473871616655468
Mark Reybrouck and Tuomas Eerola. 2017. Music and its inductive power: a psychobiological and evolutionary approach to musical emotions. Frontiers in Psychology 8 (2017), 242694.
Richard M Ryan and Edward L Deci. 2001. On happiness and human potentials: A review of research on hedonic and eudaimonic well-being. Annual review of psychology 52, 1 (2001), 141–166
Richard M Ryan and Edward L Deci. 2017. Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford publications.
Ilja Salakka, Anni Pitkäniemi, Emmi Pentikäinen, Kari Mikkonen, Pasi Saari, Petri Toiviainen, and Teppo Särkämö. 2021. What makes music memorable? Relationships between acoustic musical features and music-evoked emotions and memories in older adults. PloS one 16, 5 (2021), e0251692.
Kathryn Lupacchino Schmidt. 2018. Meaningful Music Visualizations. Ph. D. Dissertation. Purdue University.
Alexa Siu, Gene S-H Kim, Sile O’Modhrain, and Sean Follmer. 2022. Supporting Accessible Data Visualization Through Audio Data Narratives. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 476, 19 pages. DOI: 10.1145/3491102.3517678
R. Stewart, P. Kudumakis, and M. Sandler. 2010. Interactive Music Applications and Standards. (2010), 20–30. DOI: 10.1007/978-3-642-23126-1_2
Qu Tian. 2007. A Survey on Music Visualization. Computer Science (2007).
Yubo Wang, Fengzhou Pan, Danni Liu, and Jiaxiong Hu. 2023. Music-to-Facial Expressions: Emotion-Based Music Visualization for the Hearing Impaired. Proceedings of the AAAI Conference on Artificial Intelligence 37, 13 (Sep. 2023), 16096–16102. DOI: 10.1609/aaai.v37i13.26912
Yingfang Zhang, Yi Pan, and Junren Zhou. 2018. Study on application of audio visualization in new media art. In Journal of Physics: Conference Series, Vol. 1098. IOP Publishing, 012003.
1976. Atari Video Music
2018. Design Thinking. [link]
2019. Beat Saber. [link]
Horhanna Almeida, Giordano Cabral, and Rute Moura. 2021. Design process and rapid prototyping of animated music visualizations. In Anais do XVIII Simpósio Brasileiro de Computação Musical. SBC, 114–120.
Robert J Brown. 1978. Audio activated video display. US Patent 4,081,829.
Gabriel Dias Cantareira, Luis Gustavo Nonato, and Fernando V. Paulovich. 2016. MoshViz: A Detail+Overview Approach to Visualize Music Elements. IEEE Transactions on Multimedia 18, 11 (2016), 2238–2246. DOI: 10.1109/TMM.2016.2614226
Mona Lisa Chanda and Daniel J Levitin. 2013. The neurochemistry of music. Trends in cognitive sciences 17, 4 (2013), 179–193.
Marshall Chasin. 2003. Music and hearing aids. The Hearing Journal 56, 7 (2003), 36–38.
Chuer Chen, Nan Cao, Jiani Hou, Yi Guo, Yulei Zhang, and Yang Shi. 2023. MusicJam: Visualizing Music Insights via Generated Narrative Illustrations. arXiv preprint arXiv:2308.11329 (2023).
Alan S Cowen, Xia Fang, Disa Sauter, and Dacher Keltner. 2020. What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures. Proceedings of the National Academy of Sciences 117, 4 (2020), 1924–1934.
Caroline Cuny, M. Fornerino, and A. Helme-Guizon. 2015. Can music improve e-behavioral intentions by enhancing consumers’ immersion and experience? Inf. Manag. 52 (2015), 1025–1034. DOI: 10.1016/j.im.2015.07.009
Roberto De Prisco, Delfina Malandrino, Donato Pirozzi, Gianluca Zaccagnino, and Rocco Zaccagnino. 2018. Evaluation study of visualisations for harmonic analysis of 4-part music. In 2018 22nd International Conference Information Visualisation (IV). IEEE, 484–489.
Pieter Desmet. 2018. Measuring emotion: Development and application of an instrument to measure emotional responses to products. Funology 2: From Usability to Enjoyment (2018), 391–404.
Ed Diener and Micaela Y Chan. 2011. Happy people live longer: Subjective well-being contributes to health and longevity. Applied Psychology: Health and Well-Being 3, 1 (2011), 1–43
David W Fourney and Deborah I Fels. 2009. Creating access to music through visualization. In 2009 ieee toronto international conference science and technology for humanity (tic-sth). IEEE, 939–944.
Susan Hallam and Raymond MacDonald. 2013. Introduction: Perspectives on the power of music. , 83–86 pages.
Marc Hassenzahl. 2008. User experience (UX): towards an experiential perspective on product quality. (2008), 11–15. DOI: 10.1145/1512714.1512717
Rumi Hiraga, Fumiko Watanabe, and Issei Fujishiro. 2002. Music learning through visualization. In Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC 2002. Proceedings. IEEE, 101–108.
Jukka Holm and Harri Siirtola. 2012. A Comparison of Methods for Visualizing Musical Genres. In 2012 16th International Conference on Information Visualisation. 636–645. DOI: 10.1109/IV.2012.107
Felicia A Huppert. 2009. Psychological well-being: Evidence regarding its causes and consequences. Applied psychology: health and well-being 1, 2 (2009), 137–164
A. Hwang and Jeeyun Oh. 2020. Interacting with background music engages E-Customers more: The impact of interactive music on consumer perception and behavioral intention. Journal of Retailing and Consumer Services 54 (2020), 101928. DOI: 10.1016/j.jretconser.2019.101928
Patrik N Juslin and Petri Laukka. 2004. Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of new music research 33, 3 (2004), 217–238.
Patrik N Juslin and Daniel Västfjäll. 2008. Emotional responses to music: The need to consider underlying mechanisms. Behavioral and brain sciences 31, 5 (2008), 559–575.
Hiroharu Kamioka, Kiichiro Tsutani, Minoru Yamada, Hyuntae Park, Hiroyasu Okuizumi, Koki Tsuruoka, Takuya Honda, Shinpei Okada, Sang-Jun Park, Jun Kitayuguchi, et al. 2014. Effectiveness of music therapy: a summary of systematic reviews based on randomized controlled trials of music interventions. Patient preference and adherence (2014), 727–754.
Corey LM Keyes. 2007. Promoting and protecting mental health as flourishing: a complementary strategy for improving national mental health. American psychologist 62, 2 (2007), 95.
Stefan Koelsch. 2010. Towards a neural basis of music-evoked emotions. Trends in cognitive sciences 14, 3 (2010), 131–137.
Stefan Koelsch. 2014. Brain Correlate of music-evoked emotions. Nature reviews. Neuroscience 15 (02 2014), 170–180. DOI: 10.1038/nrn3666
Stefan Koelsch. 2015. Music-evoked emotions: principles, brain correlates, and implications for therapy. Annals of the New York Academy of Sciences 1337, 1 (2015), 193–201.
Yunli Lee and Revina Nur Fathia. 2016. Interactive music visualization for music player using processing. In 2016 22nd International Conference on Virtual System & Multimedia (VSMM). IEEE, 1–4.
Hugo B Lima, Carlos GR Dos Santos, and Bianchi S Meiguins. 2021. A survey of music visualization techniques. ACM Computing Surveys (CSUR) 54, 7 (2021), 1–29.
Scott D Lipscomb and Roger A Kendall. 1994. Perceptual judgement of the relationship between musical and visual components in film. Psychomusicology: A Journal of Research in Music Cognition 13, 1-2 (1994), 60.
M. Matessino. 1999. Letter in response to ’A Study of Jaws’ incisive overture to close off the century’. Internet. [link] [Accessed on March 4, 2009].
David Miranda, Carmen Li, and Ticianne Darin. 2021. UES-Br: Translation and Cross-Cultural Adaptation of the User Engagement Scale for Brazilian Portuguese. Proceedings of the ACM on Human-Computer Interaction 5 (10 2021), 1–22. 10.1145/3474705
Luminita Heliana Munteanu. 2012. Musical culture, a finality of musical education. Procedia-Social and Behavioral Sciences 46 (2012), 4195–4199.
D. Pereira and T. Chambel. 2023. Enhancing Emotional Awareness and Regulation in Movies and Music Based on Personality. Proceedings of the 2023 ACM International Conference on Interactive Media Experiences (2023). DOI: 10.1145/3573381.3596462
Caio Pereira Nunes and Ticianne Darin. 2024. Cross-cultural adaptation of the Intrinsic Motivation Inventory Task Evaluation Questionnaire into Brazilian Portuguese. 1–11. DOI: 10.1145/3638067.3638083
Ramesh Pokharel. 2020. Exploring the Different Roles and Aspects of Music. Molung Educational Frontier (2020), 125–133.
Michael Pouris and Deborah I Fels. 2012. Creating an entertaining and informative music visualization. In International Conference on Computers for Handicapped Persons. Springer, 451–458.
Jennifer Preece, Yvonne Rogers, and Helen Sharp. 2013. Design de Interação: além da interação humano-computador. Bookman.
Roberto De Prisco, Delfina Malandrino, Donato Pirozzi, Gianluca Zaccagnino, and Rocco Zaccagnino. 2017. Understanding the structure of musical compositions: Is visualization an effective approach? Information Visualization 16, 2 (2017), 139–152. DOI: 10.1177/1473871616655468
Mark Reybrouck and Tuomas Eerola. 2017. Music and its inductive power: a psychobiological and evolutionary approach to musical emotions. Frontiers in Psychology 8 (2017), 242694.
Richard M Ryan and Edward L Deci. 2001. On happiness and human potentials: A review of research on hedonic and eudaimonic well-being. Annual review of psychology 52, 1 (2001), 141–166
Richard M Ryan and Edward L Deci. 2017. Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford publications.
Ilja Salakka, Anni Pitkäniemi, Emmi Pentikäinen, Kari Mikkonen, Pasi Saari, Petri Toiviainen, and Teppo Särkämö. 2021. What makes music memorable? Relationships between acoustic musical features and music-evoked emotions and memories in older adults. PloS one 16, 5 (2021), e0251692.
Kathryn Lupacchino Schmidt. 2018. Meaningful Music Visualizations. Ph. D. Dissertation. Purdue University.
Alexa Siu, Gene S-H Kim, Sile O’Modhrain, and Sean Follmer. 2022. Supporting Accessible Data Visualization Through Audio Data Narratives. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 476, 19 pages. DOI: 10.1145/3491102.3517678
R. Stewart, P. Kudumakis, and M. Sandler. 2010. Interactive Music Applications and Standards. (2010), 20–30. DOI: 10.1007/978-3-642-23126-1_2
Qu Tian. 2007. A Survey on Music Visualization. Computer Science (2007).
Yubo Wang, Fengzhou Pan, Danni Liu, and Jiaxiong Hu. 2023. Music-to-Facial Expressions: Emotion-Based Music Visualization for the Hearing Impaired. Proceedings of the AAAI Conference on Artificial Intelligence 37, 13 (Sep. 2023), 16096–16102. DOI: 10.1609/aaai.v37i13.26912
Yingfang Zhang, Yi Pan, and Junren Zhou. 2018. Study on application of audio visualization in new media art. In Journal of Physics: Conference Series, Vol. 1098. IOP Publishing, 012003.
Published
2024-11-07
How to Cite
NUNES, Caio; REINBOLD, Isabelle; CASTRO, Mariana; DARIN, Ticianne.
Thunder: A Design Process to Build Emotionally Engaging Music Visualizations. In: BRAZILIAN SYMPOSIUM ON HUMAN FACTORS IN COMPUTATIONAL SYSTEMS (IHC), 23. , 2024, Brasília/DF.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 420-434.
