AI-Driven Emotion Prediction in Olfactory VR: A Multimodal Biofeedback Approach Using Russell’s Circumplex Model
Resumo
Despite its importance for immersion in virtual reality (VR), multisensory integration, particularly olfaction, remains underexplored. This study addresses this by developing AI computational models that analyze tri-channel physiological biofeedback (eye tracking, ECG, and GSR) to quantify the sense of presence in VR. Our methodology focuses on interpreting emotional and attentional states from these multimodal signals using supervised learning and explainable AI. Validated using the VREED dataset, which includes synchronized physiological and behavioral data from 360° immersive experiences, this work lays the groundwork for future integration into the SOFIA System. SOFIA is an olfactory-enhanced VR platform that synchronizes scent delivery with user gaze and depth. By enabling real-time emotion classification and presence estimation from physiological parameters, this research empowers the development of adaptive, personalized VR systems. This, in turn, opens new frontiers for more immersive, therapeutically effective, and tailored multisensory environments for digital well-being.
Palavras-chave:
Olfactory Virtual Reality, Enactive Smelling, Presence, Emotion Recognition, Biofeedback
Referências
S. Niedenthal, W. Fredborg, P. Lunden, M. Ehrndal, and J. K. Olofsson. A graspable olfactory display for virtual reality. Int. J. Hum.-Comput. Stud. 169:102928, 2023.
M. Slater, D. Banakou, A. Beacco, J. Gallego, F. Macia-Varela, and R. Oliva. A separate reality An update on place illusion and plausibility in virtual reality. Front. Virtual Real. 3:914392, 2022.
G. Riva, B. K. Wiederhold, and F. Mantovani. Neuroscience of virtual reality From virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. 22(1):82–96, 2019.
I. B. Adhanom, P. MacNeilage, and E. Folmer. Eye tracking in virtual reality A broad review of applications and challenges. Virtual Reality, 27(2):1481–1505, 2023.
J. Kim, J. Park, and J. Park. Development of a statistical model to classify driving stress levels using galvanic skin responses. Hum. Factors Ergonom. Manuf. Serv. Ind. 30(5):321–328, 2020.
Z. Ahmad, S. Rabbani, M. R. Zafar, S. Ishaque, S. Krishnan, and N. Khan. Multilevel stress assessment from ECG in a virtual reality environment using multimodal fusion. IEEE Sensors J. 23(23):29559–29570, 2023.
A. Steed, D. Archer, L. Izzouzi, N. Numan, K. Shapiro, D. Swapp, D. Lammiman, and R. W. Lindeman. Immersive competence and immersive literacy Exploring how users learn about immersive experiences. Front. Virtual Real. 4, 2023.
J. Russell. A circumplex model of affect. J. Pers. Soc. Psychol. 39:1161–1178, 1980.
S. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874, 2017.
M. M. Bradley and P. J. Lang. Measuring emotion The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry, 25(1):49–59, 1994.
M. Yildirim, A. Globa, O. Gocer, and A. Brambilla. Digital smell technologies for the built environment Evaluating human responses to multisensory stimuli in immersive virtual reality. Build. Environ. 271:112608, 2025.
N. S. Archer, A. Bluff, A. Eddy, C. K. Nikhil, N. Hazell, D. Frank, and A. Johnston. Odour enhances the sense of presence in a virtual reality environment. PLoS ONE, 17(3):1–20, 2022.
N. Dozio, E. Maggioni, D. Pittera, A. Gallace, and M. Obrist. May I smell your attention Exploration of smell and sound for visuospatial attention in virtual reality. Front. Psychol. 12:671470, 2021.
M. K. S. Lopes and T. H. Falk. Audio-visual-olfactory immersive digital nature exposure for stress and anxiety reduction A systematic review on systems, outcomes, and challenges. Front. Virtual Real. 5, 2024.
J. L. Contreras-Vidal, K. L. Kerick, S. M. Cao, R. G. Reynolds, M. M. G. McDowell, and D. J. McCauley. Physiological sensor technologies in workload estimation A review. IEEE Trans. Syst. Man Cybern. C Appl. Rev. 37(5):578–591, 2007.
L. Tabbaa, R. Searle, S. M. Bafti, M. M. Hossain, J. Intarasisrisawat, M. Glancy, and C. S. Ang. VREED Virtual reality emotion recognition dataset using eye tracking and physiological measures. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5(4):178:1–178:20, 2021.
Y. Pratviel, P. Bouny, and V. Deschodt-Arsac. Immersion in a relaxing virtual reality environment is associated with similar effects on stress and anxiety as heart rate variability biofeedback. Front. Virtual Real. 5, 2024.
M. Bordegoni and M. Carulli. Olfactory Virtual Reality Theory, Design, and Applications. Springer, Cham, 2021.
M. Obrist, C. Velasco, and C. T. Vi, eds. Virtual Taste and Smell Technologies for Multisensory Internet and Virtual Reality. Springer, Cham, 2017.
A. Clark. Being there Putting brain, body, and world together again. MIT Press, Cambridge, 1999.
A. Noe. Action in perception. MIT Press, Cambridge, 2004.
J. K. Olofsson, I. Ekstrom, J. Lindstrom, E. Syrjanen, A. Stigsdotter-Neely, L. Nyberg, S. Jonsson, and M. Larsson. Smell-based memory training Evidence of olfactory learning and transfer to the visual domain. Chem. Senses, 45(7):593–600, 2020.
M. F. B. da Silva, J. G. da Silva Junior, E. Souza, J. P. C. G. Fernandes, C. B. da Silva, M. Mottin, A. R. Galvao Filho, and C. H. Andrade. SOFIA Advancements in olfactory stimuli in virtual reality for immersive multisensory experiences and potential applications in the Industry 5.0. Proc. IEEE Int. Conf. Artif. Intell. eXtended Virtual Reality (AIxVR), 419–424, 2025.
Y. Kim, H. Kim, and J. Park. Arousal evaluation of VR affective scenes based on HR and SAM. Proc. ACM Symp. Virtual and Augmented Reality (SVR), 2020, 1–5.
J. Lee, M. Choi, and S. Kim. Applying SAM to evaluate the affective arousal effects of VR games. Proc. ACM Symp. Virtual and Augmented Reality (SVR), 2019, 1–7.
N. Marin-Morales, J. Llinares, A. Guixeres, and M. Alcaniz. Virtual reality for emotion elicitation A review. Front. Psychol. 11:451, 2020.
M. F. Brito da Silva, I. H. Sanches, P. K. Diniz, R. M. de Carvalho, F. L. Feitosa, A. C. de A. Barros, J. P. C. G. Fernandes, C. B. da Silva, J. G. da Silva Junior, A. R. Galvao Filho, and C. H. Andrade. Creating an olfactory mask dataset using computer vision and LLMs for virtual reality environments. Proc. IEEE Int. Conf. Artif. Intell. eXtended Virtual Reality (AIxVR), 459–463, 2025.
M. F. Brito da Silva, P. K. Diniz, R. M. de Carvalho, M. C. A. Neto, T. H. B. da Silva, A. C. de A. Barros, Y. T. S. Ferreira, F. L. Feitosa, J. P. C. G. Fernandes, C. B. da Silva, J. G. da Silva Junior, A. G. de Siqueira, A. R. Galvao Filho, and C. H. Andrade. SOFIA Advancements in olfactory stimuli in virtual reality for immersive multisensory experiences and potential applications in Industry 5.0. Proc. IEEE Int. Conf. Artif. Intell. eXtended Virtual Reality (AIxVR), 419–424, 2025.
M. Slater, D. Banakou, A. Beacco, J. Gallego, F. Macia-Varela, and R. Oliva. A separate reality An update on place illusion and plausibility in virtual reality. Front. Virtual Real. 3:914392, 2022.
G. Riva, B. K. Wiederhold, and F. Mantovani. Neuroscience of virtual reality From virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. 22(1):82–96, 2019.
I. B. Adhanom, P. MacNeilage, and E. Folmer. Eye tracking in virtual reality A broad review of applications and challenges. Virtual Reality, 27(2):1481–1505, 2023.
J. Kim, J. Park, and J. Park. Development of a statistical model to classify driving stress levels using galvanic skin responses. Hum. Factors Ergonom. Manuf. Serv. Ind. 30(5):321–328, 2020.
Z. Ahmad, S. Rabbani, M. R. Zafar, S. Ishaque, S. Krishnan, and N. Khan. Multilevel stress assessment from ECG in a virtual reality environment using multimodal fusion. IEEE Sensors J. 23(23):29559–29570, 2023.
A. Steed, D. Archer, L. Izzouzi, N. Numan, K. Shapiro, D. Swapp, D. Lammiman, and R. W. Lindeman. Immersive competence and immersive literacy Exploring how users learn about immersive experiences. Front. Virtual Real. 4, 2023.
J. Russell. A circumplex model of affect. J. Pers. Soc. Psychol. 39:1161–1178, 1980.
S. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874, 2017.
M. M. Bradley and P. J. Lang. Measuring emotion The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry, 25(1):49–59, 1994.
M. Yildirim, A. Globa, O. Gocer, and A. Brambilla. Digital smell technologies for the built environment Evaluating human responses to multisensory stimuli in immersive virtual reality. Build. Environ. 271:112608, 2025.
N. S. Archer, A. Bluff, A. Eddy, C. K. Nikhil, N. Hazell, D. Frank, and A. Johnston. Odour enhances the sense of presence in a virtual reality environment. PLoS ONE, 17(3):1–20, 2022.
N. Dozio, E. Maggioni, D. Pittera, A. Gallace, and M. Obrist. May I smell your attention Exploration of smell and sound for visuospatial attention in virtual reality. Front. Psychol. 12:671470, 2021.
M. K. S. Lopes and T. H. Falk. Audio-visual-olfactory immersive digital nature exposure for stress and anxiety reduction A systematic review on systems, outcomes, and challenges. Front. Virtual Real. 5, 2024.
J. L. Contreras-Vidal, K. L. Kerick, S. M. Cao, R. G. Reynolds, M. M. G. McDowell, and D. J. McCauley. Physiological sensor technologies in workload estimation A review. IEEE Trans. Syst. Man Cybern. C Appl. Rev. 37(5):578–591, 2007.
L. Tabbaa, R. Searle, S. M. Bafti, M. M. Hossain, J. Intarasisrisawat, M. Glancy, and C. S. Ang. VREED Virtual reality emotion recognition dataset using eye tracking and physiological measures. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5(4):178:1–178:20, 2021.
Y. Pratviel, P. Bouny, and V. Deschodt-Arsac. Immersion in a relaxing virtual reality environment is associated with similar effects on stress and anxiety as heart rate variability biofeedback. Front. Virtual Real. 5, 2024.
M. Bordegoni and M. Carulli. Olfactory Virtual Reality Theory, Design, and Applications. Springer, Cham, 2021.
M. Obrist, C. Velasco, and C. T. Vi, eds. Virtual Taste and Smell Technologies for Multisensory Internet and Virtual Reality. Springer, Cham, 2017.
A. Clark. Being there Putting brain, body, and world together again. MIT Press, Cambridge, 1999.
A. Noe. Action in perception. MIT Press, Cambridge, 2004.
J. K. Olofsson, I. Ekstrom, J. Lindstrom, E. Syrjanen, A. Stigsdotter-Neely, L. Nyberg, S. Jonsson, and M. Larsson. Smell-based memory training Evidence of olfactory learning and transfer to the visual domain. Chem. Senses, 45(7):593–600, 2020.
M. F. B. da Silva, J. G. da Silva Junior, E. Souza, J. P. C. G. Fernandes, C. B. da Silva, M. Mottin, A. R. Galvao Filho, and C. H. Andrade. SOFIA Advancements in olfactory stimuli in virtual reality for immersive multisensory experiences and potential applications in the Industry 5.0. Proc. IEEE Int. Conf. Artif. Intell. eXtended Virtual Reality (AIxVR), 419–424, 2025.
Y. Kim, H. Kim, and J. Park. Arousal evaluation of VR affective scenes based on HR and SAM. Proc. ACM Symp. Virtual and Augmented Reality (SVR), 2020, 1–5.
J. Lee, M. Choi, and S. Kim. Applying SAM to evaluate the affective arousal effects of VR games. Proc. ACM Symp. Virtual and Augmented Reality (SVR), 2019, 1–7.
N. Marin-Morales, J. Llinares, A. Guixeres, and M. Alcaniz. Virtual reality for emotion elicitation A review. Front. Psychol. 11:451, 2020.
M. F. Brito da Silva, I. H. Sanches, P. K. Diniz, R. M. de Carvalho, F. L. Feitosa, A. C. de A. Barros, J. P. C. G. Fernandes, C. B. da Silva, J. G. da Silva Junior, A. R. Galvao Filho, and C. H. Andrade. Creating an olfactory mask dataset using computer vision and LLMs for virtual reality environments. Proc. IEEE Int. Conf. Artif. Intell. eXtended Virtual Reality (AIxVR), 459–463, 2025.
M. F. Brito da Silva, P. K. Diniz, R. M. de Carvalho, M. C. A. Neto, T. H. B. da Silva, A. C. de A. Barros, Y. T. S. Ferreira, F. L. Feitosa, J. P. C. G. Fernandes, C. B. da Silva, J. G. da Silva Junior, A. G. de Siqueira, A. R. Galvao Filho, and C. H. Andrade. SOFIA Advancements in olfactory stimuli in virtual reality for immersive multisensory experiences and potential applications in Industry 5.0. Proc. IEEE Int. Conf. Artif. Intell. eXtended Virtual Reality (AIxVR), 419–424, 2025.
Publicado
30/09/2025
Como Citar
SILVA JUNIOR, Jonas G. Da et al.
AI-Driven Emotion Prediction in Olfactory VR: A Multimodal Biofeedback Approach Using Russell’s Circumplex Model. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 27. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 253-262.
