Quantifying Multisensory Immersive Experiences using Wearables: Is (Stimulating) More (Senses) Always Merrier?

• Belmir De Jesus Jr. University of Quebec
• Marilia Lopes University of Quebec
• Marc-Antoine Moinnereau University of Quebec
• Reza A. Gougeh University of Quebec
• Olivier M. Rosanne University of Quebec
• Walter Schubert University of Quebec / UFCSPA
• Alcyr A. Oliveira University of Quebec / UFCSPA
• Tiago H. Falk University of Quebec

DOI: https://doi.org/10.5753/sensoryx.2022.20001

Abstract

Virtual reality applications are on the rise and touching numerous domains, including healthcare, training, and gaming, to name a few. Existing experiences, however, are not fully immersive, as only two senses (audio-visual) are stimulated. To overcome this limitation, olfactory and haptic devices are emerging, thus making multisensory immersive experiences a reality. To date, however, little is known about the impact that each stimulated sense has on the overall experience, as well as on the user’s sense of e.g., realism, immersion, and engagement. In this pilot, we aim to answer this question. Using a multisensory pod, sixteen participants were immersed in a 2.5-minute virtual world, where smells, vibroacoustic, and somatosensory stimuli (i.e., wind and heat) were presented, in addition to 360-degree video and surround sound. Using two wearable devices, we kept track of the user’s heart rate, breathing rate, skin temperature, blood volume pulse, and electrodermal activity while they were immersed. In this paper, we report the impact that stimulating different senses had on the users’ overall experience, sense of presence, immersion, realism, flow, cybersickness, and emotional states, both subjectively, as well as objectively using features extracted from the wearable devices.

Keywords: Virtual reality, quality of experience, multisensory, olfaction, haptics

References

John G Apostolopoulos, Philip A Chou, Bruce Culbertson, Ton Kalker, Mitchell D Trott, and Susie Wee. 2012. The road to immersive communication. Proc. IEEE 100, 4 (2012), 974–990.

Raymundo Cassani, Marc-Antoine Moinnereau, Liviu Ivanescu, Olivier Rosanne, and Tiago H Falk. 2020. Neural Interface instrumented virtual reality headsets: Toward next-generation immersive applications. IEEE Systems, Man, and Cybernetics Magazine 6, 3 (2020), 20–28.

Ho-Seung Cha and Chang-Hwan Im. 2022. Performance enhancement of facial electromyogram-based facial-expression recognition for social virtual reality applications using linear discriminant analysis adaptation. Virtual Reality 26, 1 (2022), 385–398.

Ioan-Sorin Comşa, Estêvão Bissoli Saleme, Alexandra Covaci, Gebremariam Mesfin Assres, Ramona Trestian, Celso AS Santos, and Gheorghita Ghinea. 2019. Do I smell coffee? The tale of a 360 mulsemedia experience. IEEE MultiMedia 27, 1 (2019), 27–36.

Gheorghita Ghinea, Christian Timmerer, Weisi Lin, and Stephen R Gulliver. 2014. Mulsemedia: State of the art, perspectives, and challenges. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 11, 1s (2014), 1–23.

Eric Krokos and Amitabh Varshney. 2022. Quantifying VR cybersickness using EEG. Virtual Reality 26, 1 (2022), 77–89.

MarketsAndMarkets. 2020. Virtual Reality Market with COVID-19 Impact Analysis by Offering, Technology, Device Type, Application and Geography - Global Forecast to 2025. Technical Report. Markets and Markets.

Gebremariam Mesfin, Estevao Bissoli Saleme, Oluwakemi A Ademoye, Elahe Kani-Zabihi, Celso AS Santos, and Gheorghita Ghinea. 2020. Less is (just as good as) more-an investigation of odor intensity and hedonic valence in mulsemedia qoe using heart rate and eye tracking. IEEE Transactions on Multimedia 23 (2020), 1095–1105.

Marc-Antoine Moinnereau, Alcyr Oliveira, and Tiago H Falk. 2020. Saccadic Eye Movement Classification Using ExG Sensors Embedded into a Virtual Reality Headset. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3494–3498.

Marc-Antoine Moinnereau, Alcyr Olivera, and Tiago H. Falk. 2022. Measuring Human Influential Factors During VR Gaming at Home: Towards Optimized Per-User Gaming Experiences. In International Conference on Applied Human Factors and Ergonomics.

Diego Monteiro, Hai-Ning Liang, Andrew Abel, Nilufar Bahaei, and Rita de Cassia Monteiro. 2018. Evaluating engagement of virtual reality games based on first and third person perspective using EEG and subjective metrics. In 2018 IEEE international conference on artificial intelligence and virtual reality (AIVR). Ieee, 53–60.

David Narciso, Miguel Melo, José Vasconcelos-Raposo, and Maximino Bessa. 2020. The impact of olfactory and wind stimuli on 360 videos using head-mounted displays. ACM Transactions on Applied Perception (TAP) 17, 1 (2020), 1–13.

Andrew Perkis, Christian Timmerer, Sabina Barakovic, et al. 2020. QUALINET white paper on definitions of immersive media experience (IMEx). arXiv preprint arXiv:2007.07032 (2020).

Christoph Tremmel, Christian Herff, Tetsuya Sato, Krzysztof Rechowicz, Yusuke Yamani, and Dean J Krusienski. 2019. Estimating cognitive workload in an interactive virtual reality environment using EEG. Frontiers in human neuroscience 13 (2019), 401.