Improving User Evaluations in Multisensory Environments through Physiological Signals

• Aleph Campos da Silveira UFES
• Celso Alberto Saibel Santos UFES

DOI: https://doi.org/10.5753/webmedia_estendido.2026.34206

Resumo

Mulsemedia systems enrich audiovisual content with effects such as wind, temperature, smell, and haptics, but their evaluation still relies mainly on retrospective questionnaires. In immersive experiences, these measures can aggregate distinct moments, interrupt presence, and overlook unreported reactions. This doctoral thesis investigates how physiological signals can complement subjective UX and QoE measures. Its central contribution is a six-stage, technology-independent methodology integrating study planning, participant preparation, synchronized physiological and event-data acquisition, post-session feedback, and multimodal analysis. Informed by a systematic review, the methodology was examined through four studies involving thermal-wind video, audiovisual asynchrony in 360-degree video, olfactory augmentation, and a firefighter serious game. Results show that immersive exploration can reduce the reported salience of audiovisual offsets, congruent scent can stabilize ratings under asynchrony, and static scent delivery can lose coherence when VR users move. Physiological measurements add continuous temporal evidence but require triangulation with self-reports and system events because activation does not uniquely identify emotion. The thesis contributes a replicable evaluation methodology, findings on crossmodal coherence, and practical guidelines for more reliable multisensory systems.

Palavras-chave: mulsemedia, physiological signals, user experience, quality of experience, virtual reality

Referências

Aleph Campos da Silveira, Mariane Lima de Souza, Gheorghita Ghinea, and Celso Alberto Saibel Santos. 2024. Physiological Data for User Experience and Quality of Experience: A Systematic Review (2018–2022). International Journal of Human–Computer Interaction 0, 0 (2024), 1–30. DOI: 10.1080/10447318.2024.2311972

Aleph Campos da Silveira, Eduardo C. Rodrigues, Estevao B. Saleme, Alexandra Covaci, Gheorghita Ghinea, and Celso A. S. Santos. 2023. Thermal and wind devices for multisensory human-computer interaction: an overview. Multimedia Tools and Applications 82, 22 (March 2023), 34485–34512. DOI: 10.1007/s11042-023-14672-y

Aleph Campos da Silveira, Fotios Spyridonis, Roope Rosaimo, Alexandra Covaci, Gheorghita Ghinea, and Celso Alberto Saibel Santos. 2024. On Perceived AV Synchronization in 360° Multimedia. IEEE MultiMedia 31, 3 (2024), 7–17. DOI: 10.1109/MMUL.2024.3415116

Aleph Campos Silveira and Celso Alberto SAibel Santos. 2022. Ongoing Challenges of Evaluating Mulsemedia QoE. In 2nd Workshop on Multisensory Experiences (SensoryX’22) (Aveiro). SBC, Porto Alegre, RS, Brasil. DOI: 10.5753/sensoryx.2022.20005