Live Feedback for Immersive Music Performances - A Case Study
Abstract
This paper presents the architecture of an Extended Reality (XR) system for real-time streaming and spatial rendering of live music performances, along with a comprehensive setup for the acquisition of multichannel audio and three-dimensional performer motion data. The system enables remote users to experience live performances in immersive virtual environments, where each musician is represented by an avatar. Audio input is captured via a digital mixing console, enabling the transmission of synchronized multitrack audio streams using low-latency networking protocols. These streams are spatially rendered within a 3D scene using advanced audio spatialization techniques integrated into standard real-time engines such as Unity. The system also supports bidirectional interaction: audience-generated audio feedback (e.g., cheering, clapping, or singing) is captured, spatially processed, and reintroduced into the performance environment, enhancing performer-audience engagement through immersive, real-time crowd response. The architecture was set-up and initially tested in a real concert environment at the Rockstadt Club in Bras, ov, Romania, with the occasion of a live blues concert in 2024.
Keywords:
Extended Reality, Multi-Channel Audio, Virtual Environment
References
2024. Soundjack: low-latency p2p and server streaming application. Online Resource. Available at [link].
Patrick Cairns. 2021. VIIVA-NMP Audio System: The design of a low latency and naturally interactive Ambisonic audio system for Immersive Network Music Performance. Master’s thesis. University of York.
Patrick Cairns, Helena Daffern, and Gavin Kearney. 2024. Investigation of Server–Based Spatial Audio for Metaverse Concert Distribution. In 2024 IEEE 5th International Symposium on the Internet of Sounds (IS2). 1–8. DOI: 10.1109/IS262782.2024.10704097
Alexander Carôt and Christian Werner. 2007. Network music performance problems, approaches and perspectives. In Proceedings of the “Music in the Global Village”-Conference, Budapest, Hungary, Vol. 162. 23–10.
Luca Comanducci. 2023. Intelligent Networked Music Performance Experiences. Springer International Publishing, Cham, 119–130. DOI: 10.1007/978-3-031-15374-7_10
Carlo Drioli, Claudio Allocchio, and Nicola Buso. 2013. Networked Performances and Natural Interaction via LOLA: Low Latency High Quality A/V Streaming System. In Information Technologies for Performing Arts, Media Access, and Entertainment, Paolo Nesi and Raffaella Santucci (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 240–250.
dürre jan, werner norbert, hämäläinen seppo, lindfors oscar, koistinen janne, saarenmaa miro, and hupke robert. 2022. in-depth latency and reliability analysis of a networked music performance over public 5g infrastructure. journal of the audio engineering society 10621 (october 2022).
Melanie Fernandes, Nicole Mallmann, and Sangwoo Shin. 2024. The Rise of Augmented Reality in Live Music Events: The Cases of Snapchat and Gorillaz. Business Communication Research and Practice 7, 1 (2024), 58–63. DOI: 10.22682/bcrp.2024.7.1.58
Andrea F. Genovese, Marta Gospodarek, Zack Nguyen, Robert Pahle, and Agnieszka Roginska. 2024. Locally Adapted Immersive Environmentsfor Distributed Music Performances in Mixed Reality. In 2024 IEEE 5th International Symposium on the Internet of Sounds (IS2). 1–10. DOI: 10.1109/IS262782.2024.10704217
Stefano Giacomelli, Carlo Centofanti, Jose Santos, Mauro Galbiati, Tiziano Salvi, Fabio Graziosi, and Claudia Rinaldi. 2024. Remote Immersive Audio Production: State of the Art Implementation, Challenges, and Improvements. 1–10. DOI: 10.1109/IS262782.2024.10704192
Google Inc. 2018. Resonance Audio: Spatial Audio for VR, AR, and 360-degree video. Online Resource. Available at [link].
Michael Gurevich, Adam Fyans, and Paul Stapleton. 2004. JamSpace: Interactive Music Collaboration on the Web. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME). 109–112.
Rob Hamilton. 2023. Real-time musical performance across and within extended reality environments. The Journal of the Acoustical Society of America 153 (03 2023), A35–A35. DOI: 10.1121/10.0018060
Rory Hoy and Doug Van Nort. 2021. A Technological and Methodological Ecosystem for Dynamic Virtual Acoustics in Telematic Performance Contexts. 169–174. DOI: 10.1145/3478384.3478425
Bartłomiej Mróz, Piotr Odya, Przemyslaw Danowski, and Marek Kabaciński. 2023. A commonly-accessible toolchain for live streaming music events with higher-order ambisonic audio and 4k 360 vision.
Cristina Rottondi, Chris Chafe, Claudio Allocchio, and Augusto Sarti. 2016. An Overview on Networked Music Performance Technologies. IEEE Access 4 (2016), 8823–8843. DOI: 10.1109/ACCESS.2016.2628440
Jens Schuett, Samuel Le Groux, and Pieter-Jan Maes. 2021. Reducing Latency in Remote Musical Interaction: An Edge Computing Perspective. IEEE Transactions on Multimedia 23 (2021), 4253–4264. DOI: 10.1109/TMM.2021.3051963
Barry Truax. 2012. Sound, Listening and Place: The Aesthetic Dilemma. Organised Sound 17, 3 (2012), 193–201. DOI: 10.1017/S135577181200005X
Luca Turchet, Claudia Rinaldi, Carlo Centofanti, Luca Vignati, and Cristina Rottondi. 2024. 5G-Enabled Internet of Musical Things Architectures for Remote Immersive Musical Practices. IEEE Open Journal of the Communications Society 5 (2024), 4691–4709. DOI: 10.1109/OJCOMS.2024.3407708
Valve Corporation. 2023. Steam Audio: Audio Solutions for Virtual and Augmented Reality. Online Resource. Available at [link].
Wikipedia contributors. 2024. Ambisonics. [link]. Accessed: 2024-12-11.
Wikipedia contributors. 2024. Head-Related Transfer Function. [link]. Accessed: 2024-12-11.
Elise Wilk, Daan Niermann, and Julia Hentschel. 2023. Embodiment in XR Per formances: Towards Situated, Responsive Stages. Journal of New Music Research 52, 1 (2023), 45–61. DOI: 10.1080/09298215.2023.2172846
World Wide Web Consortium (W3C). 2023. WebRTC: Real-Time Communication in Web Browsers. [link]. Accessed: 2024-12-11.
Gareth W. Young, Néill O’Dwyer, Matthew Moynihan, and Aljosa Smolic. 2022. Audience Experiences of a Volumetric Virtual Reality Music Video. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 775–781. DOI: 10.1109/VR51125.2022.00099
Patrick Cairns. 2021. VIIVA-NMP Audio System: The design of a low latency and naturally interactive Ambisonic audio system for Immersive Network Music Performance. Master’s thesis. University of York.
Patrick Cairns, Helena Daffern, and Gavin Kearney. 2024. Investigation of Server–Based Spatial Audio for Metaverse Concert Distribution. In 2024 IEEE 5th International Symposium on the Internet of Sounds (IS2). 1–8. DOI: 10.1109/IS262782.2024.10704097
Alexander Carôt and Christian Werner. 2007. Network music performance problems, approaches and perspectives. In Proceedings of the “Music in the Global Village”-Conference, Budapest, Hungary, Vol. 162. 23–10.
Luca Comanducci. 2023. Intelligent Networked Music Performance Experiences. Springer International Publishing, Cham, 119–130. DOI: 10.1007/978-3-031-15374-7_10
Carlo Drioli, Claudio Allocchio, and Nicola Buso. 2013. Networked Performances and Natural Interaction via LOLA: Low Latency High Quality A/V Streaming System. In Information Technologies for Performing Arts, Media Access, and Entertainment, Paolo Nesi and Raffaella Santucci (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 240–250.
dürre jan, werner norbert, hämäläinen seppo, lindfors oscar, koistinen janne, saarenmaa miro, and hupke robert. 2022. in-depth latency and reliability analysis of a networked music performance over public 5g infrastructure. journal of the audio engineering society 10621 (october 2022).
Melanie Fernandes, Nicole Mallmann, and Sangwoo Shin. 2024. The Rise of Augmented Reality in Live Music Events: The Cases of Snapchat and Gorillaz. Business Communication Research and Practice 7, 1 (2024), 58–63. DOI: 10.22682/bcrp.2024.7.1.58
Andrea F. Genovese, Marta Gospodarek, Zack Nguyen, Robert Pahle, and Agnieszka Roginska. 2024. Locally Adapted Immersive Environmentsfor Distributed Music Performances in Mixed Reality. In 2024 IEEE 5th International Symposium on the Internet of Sounds (IS2). 1–10. DOI: 10.1109/IS262782.2024.10704217
Stefano Giacomelli, Carlo Centofanti, Jose Santos, Mauro Galbiati, Tiziano Salvi, Fabio Graziosi, and Claudia Rinaldi. 2024. Remote Immersive Audio Production: State of the Art Implementation, Challenges, and Improvements. 1–10. DOI: 10.1109/IS262782.2024.10704192
Google Inc. 2018. Resonance Audio: Spatial Audio for VR, AR, and 360-degree video. Online Resource. Available at [link].
Michael Gurevich, Adam Fyans, and Paul Stapleton. 2004. JamSpace: Interactive Music Collaboration on the Web. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME). 109–112.
Rob Hamilton. 2023. Real-time musical performance across and within extended reality environments. The Journal of the Acoustical Society of America 153 (03 2023), A35–A35. DOI: 10.1121/10.0018060
Rory Hoy and Doug Van Nort. 2021. A Technological and Methodological Ecosystem for Dynamic Virtual Acoustics in Telematic Performance Contexts. 169–174. DOI: 10.1145/3478384.3478425
Bartłomiej Mróz, Piotr Odya, Przemyslaw Danowski, and Marek Kabaciński. 2023. A commonly-accessible toolchain for live streaming music events with higher-order ambisonic audio and 4k 360 vision.
Cristina Rottondi, Chris Chafe, Claudio Allocchio, and Augusto Sarti. 2016. An Overview on Networked Music Performance Technologies. IEEE Access 4 (2016), 8823–8843. DOI: 10.1109/ACCESS.2016.2628440
Jens Schuett, Samuel Le Groux, and Pieter-Jan Maes. 2021. Reducing Latency in Remote Musical Interaction: An Edge Computing Perspective. IEEE Transactions on Multimedia 23 (2021), 4253–4264. DOI: 10.1109/TMM.2021.3051963
Barry Truax. 2012. Sound, Listening and Place: The Aesthetic Dilemma. Organised Sound 17, 3 (2012), 193–201. DOI: 10.1017/S135577181200005X
Luca Turchet, Claudia Rinaldi, Carlo Centofanti, Luca Vignati, and Cristina Rottondi. 2024. 5G-Enabled Internet of Musical Things Architectures for Remote Immersive Musical Practices. IEEE Open Journal of the Communications Society 5 (2024), 4691–4709. DOI: 10.1109/OJCOMS.2024.3407708
Valve Corporation. 2023. Steam Audio: Audio Solutions for Virtual and Augmented Reality. Online Resource. Available at [link].
Wikipedia contributors. 2024. Ambisonics. [link]. Accessed: 2024-12-11.
Wikipedia contributors. 2024. Head-Related Transfer Function. [link]. Accessed: 2024-12-11.
Elise Wilk, Daan Niermann, and Julia Hentschel. 2023. Embodiment in XR Per formances: Towards Situated, Responsive Stages. Journal of New Music Research 52, 1 (2023), 45–61. DOI: 10.1080/09298215.2023.2172846
World Wide Web Consortium (W3C). 2023. WebRTC: Real-Time Communication in Web Browsers. [link]. Accessed: 2024-12-11.
Gareth W. Young, Néill O’Dwyer, Matthew Moynihan, and Aljosa Smolic. 2022. Audience Experiences of a Volumetric Virtual Reality Music Video. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 775–781. DOI: 10.1109/VR51125.2022.00099
Published
2025-06-03
How to Cite
AMARIE, Nicusor; FADDA, Gianluca; POPESCU, Vlad; GHENTA, Jean-Marius; MURRONI, Maurizio; SIMISCUKA, Anderson Augusto.
Live Feedback for Immersive Music Performances - A Case Study. In: ACM INTERNATIONAL CONFERENCE ON INTERACTIVE MEDIA EXPERIENCES WORKSHOPS (IMXW), 25. , 2025, Niterói/RJ.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 26-29.
DOI: https://doi.org/10.5753/imxw.2025.8520.
