SensoryX ’21 Workshop on Multisensory Experiences at ACM IMX ’21
Abstract
Multisensory experiences have been increasingly undertaken in the digital world. With the emerging interest in immersive applications (i.e. 360 videos and virtual reality), more and more researchers and practitioners are in pursuit of ways to take these experiences to the next level, adding sensations that go beyond that of seeing and hearing. This one-day workshop aims at identifying the current practices, challenges, opportunities, and limitations to be overcome in the quest to transcend the overwhelmingly bisensorial nature of digital multimedia into a multisensory one by fostering discussions on the use of olfactory, gustatory and tactile effects in digital experiences.
Keywords:
Multisensory experiences, Mulsemedia, Sensory effects, Immersive media
References
O. Ademoye and G. Ghinea. 2013. Information recall task impact in olfaction-enhanced multimedia. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 9, 3 (2013), 17.
Anas Ali Alkasasbeh and Gheorghita Ghinea. 2020. Using olfactory media cues in e-learning–perspectives from an empirical investigation. Multimedia Tools and Applications (2020), 1–23.
J. H. Chu, D. Harley, J. Kwan, M. McBride, and A. Mazalek. 2016. Sensing History: Contextualizing Artifacts with Sensory Interactions and Narrative Design. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (Brisbane, QLD, Australia) (DIS ’16). ACM, New York, NY, USA, 1294–1302. https://doi.org/10.1145/2901790.2901829
C. Claisse, D. Petrelli, N. Dulake, M. Marshall, and L. Ciolf. 2018. Multisensory interactive storytelling to augment the visit of a historical house museum. In Proceedings of the 2018 Digital Heritage International Congress. IEEE. http://shura.shu.ac.uk/22646/
A. Covaci, G. Ghinea, C-H. Lin, S-H. Huang, and J-L. Shih. 2018. Multisensory games-based learning-lessons learnt from olfactory enhancement of a digital board game. Multimedia Tools and Applications (2018), 1–19.
A. Covaci, R. Trestian, E. B. Saleme, I-S Comsa, G. Mesfn, C. A. S. Santos, and G. Ghinea. 2019. 360◦ Mulsemedia: A Way to Improve Subjective QoE in 360◦ Videos. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). ACM, New York, NY, USA, 2378–2386. https://doi.org/10.1145/3343031.3350954
A. Covaci, L. Zou, I. Tal, G-M. Muntean, and G. Ghinea. 2018. Is Multimedia Multisensorial? - A Review of Mulsemedia Systems. Comput. Surveys 51, 5, Article 91 (Sept. 2018), 35 pages. https://doi.org/10.1145/3233774
G. Ghinea and O. Ademoye. 2012. The sweet smell of success: Enhancing multimedia applications with olfaction. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 8, 1 (2012), 2.
G. Ghinea, F. Andres, and S. R. Gulliver. 2011. Multiple Sensorial Media Advances and Applications: New Developments in MulSeMedia: New Developments in MulSeMedia. Information Science Reference.
G. Ghinea, C. Timmerer, W. Lin, and S. R. Gulliver. 2014. Mulsemedia: State of the Art, Perspectives, and Challenges. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s, Article 17 (Oct. 2014), 23 pages. https://doi.org/10.1145/2617994
N. Murray, O. A. Ademoye, G. Ghinea, and G-M. Muntean. 2017. A Tutorial for Olfaction-Based Multisensorial Media Application Design and Evaluation. ACM Comp. Surveys 50, 5, Article 67 (2017), 30 pages. https://doi.org/10.1145/ 3108243
M. Obrist, C. Velasco, C. Vi, N. Ranasinghe, A. Israr, A. Cheok, C. Spence, and P. Gopalakrishnakone. 2016. Sensing the future of HCI: touch, taste, and smell user interfaces. interactions 23, 5 (2016), 40–49.
O. Petit, C. Velasco, and C. Spence. 2019. Digital Sensory Marketing: Integrating New Technologies Into Multisensory Online Experience. Journal of Interactive Marketing 45 (2019), 42 – 61. https://doi.org/10.1016/j.intmar.2018.07.004
N. Ranasinghe, P. Jain, N. Thi Ngoc Tram, K. C. R. Koh, D. Tolley, S. Karwita, L. Lien-Ya, Y. Liangkun, K. Shamaiah, C. Eason Wai Tung, C. C. Yen, and E. Y-L. Do. 2018. Season Traveller: Multisensory Narration for Enhancing the Virtual Reality Experience. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 577, 13 pages. https://doi.org/10.1145/3173574.3174151
E. B. Saleme, A. Covaci, G. Mesfn, C. A. S. Santos, and G. Ghinea. 2019. Mulsemedia DIY: A Survey of Devices and a Tutorial for Building Your Own Mulsemedia Environment. Comput. Surveys 52, 3, Article 58 (June 2019), 29 pages.
E. B. Saleme, C. A. S. Santos, R. A. Falbo, G. Ghinea, and F. Andres. 2019. MulseOnto: a Reference Ontology to Support the Design of Mulsemedia Systems. Journal of Universal Computer Science (2019), 26 pages.
E. B. Saleme, C. A. S. Santos, and G. Ghinea. 2019. Coping With the Challenges of Delivering Multiple Sensorial Media. IEEE MultiMedia 26, 2 (April 2019), 66–75. https://doi.org/10.1109/MMUL.2018.2873565
E. B. Saleme, C. A. S. Santos, and G. Ghinea. 2019. A mulsemedia framework for delivering sensory effects to heterogeneous systems. Multimedia Systems 25, 4 (01 Aug 2019), 421–447. https://doi.org/10.1007/s00530-019-00618-8
B. Serrano, R. M. Baños, and C. Botella. 2016. Virtual reality and stimulation of touch and smell for inducing relaxation: A randomized controlled trial. Computers in Human Behavior 55 (2016), 1–8.
B. S. Spencer. 2006. Incorporating the sense of smell into patient and haptic surgical simulators. IEEE Transactions on Information Technology in Biomedicine 10, 1 (Jan 2006), 168–173. https://doi.org/10.1109/TITB.2005.856851
R. J. Stone, R. Guest, P. Mahoney, D. Lamb, and C. Gibson. 2017. A ‘mixed reality’ simulator concept for future Medical Emergency Response Team training. Journal of the Royal Army Medical Corps 163, 4 (2017), 280–287. https: //doi.org/10.1136/jramc-2016-000726
Y. Sulema. 2016. Mulsemedia vs. Multimedia: State of the art and future trends. In 2016 International Conference on Systems, Signals and Image Processing (IWSSIP). 1–5. https://doi.org/10.1109/IWSSIP.2016.7502696
Irina Tal, Longhao Zou, Alexandra Covaci, Eva Ibarrola, Marilena Bratu, Gheorghita Ghinea, and Gabriel-Miro Muntean. 2019. Mulsemedia in Telecommunication and Networking Education: A Novel Teaching Approach that Improves the Learning Process. IEEE Communications Magazine 57, 11 (2019), 60–66.
Erik van der Meulen, Marina A Cidota, Stephan G Lukosch, Paulina JM Bank, Aadjan JC van der Helm, and Valentijn T Visch. 2016. A haptic serious augmented reality game for motor assessment of parkinson’s disease patients. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, 102–104.
C. Velasco and M. Obrist. 2021. Multisensory Experiences: A Primer. Frontiers in Computer Science 3 (2021), 12. https://doi.org/10.3389/fcomp.2021.614524
C. T. Vi, D. Arthur, and M. Obrist. 2018. TasteBud: Bring Taste Back into the Game. In Proceedings of the 3rd International Workshop on Multisensory Approaches to Human-Food Interaction (Boulder, CO, USA) (MHFI’18). ACM, New York, NY, USA, Article 1, 5 pages. https://doi.org/10.1145/3279954.3279955
B. Yecies. 2016. Transnational collaboration of the multisensory kind: exploiting Korean 4D cinema in China. Media International Australia 159, 1 (2016), 22–31.
L. Zou, I. Tal, A. Covaci, E. Ibarrola, G. Ghinea, and G-M. Muntean. 2017. Can Multisensorial Media Improve Learner Experience?. In Proceedings of the 8th ACM on Multimedia Systems Conference. ACM, 315–320.
Anas Ali Alkasasbeh and Gheorghita Ghinea. 2020. Using olfactory media cues in e-learning–perspectives from an empirical investigation. Multimedia Tools and Applications (2020), 1–23.
J. H. Chu, D. Harley, J. Kwan, M. McBride, and A. Mazalek. 2016. Sensing History: Contextualizing Artifacts with Sensory Interactions and Narrative Design. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (Brisbane, QLD, Australia) (DIS ’16). ACM, New York, NY, USA, 1294–1302. https://doi.org/10.1145/2901790.2901829
C. Claisse, D. Petrelli, N. Dulake, M. Marshall, and L. Ciolf. 2018. Multisensory interactive storytelling to augment the visit of a historical house museum. In Proceedings of the 2018 Digital Heritage International Congress. IEEE. http://shura.shu.ac.uk/22646/
A. Covaci, G. Ghinea, C-H. Lin, S-H. Huang, and J-L. Shih. 2018. Multisensory games-based learning-lessons learnt from olfactory enhancement of a digital board game. Multimedia Tools and Applications (2018), 1–19.
A. Covaci, R. Trestian, E. B. Saleme, I-S Comsa, G. Mesfn, C. A. S. Santos, and G. Ghinea. 2019. 360◦ Mulsemedia: A Way to Improve Subjective QoE in 360◦ Videos. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). ACM, New York, NY, USA, 2378–2386. https://doi.org/10.1145/3343031.3350954
A. Covaci, L. Zou, I. Tal, G-M. Muntean, and G. Ghinea. 2018. Is Multimedia Multisensorial? - A Review of Mulsemedia Systems. Comput. Surveys 51, 5, Article 91 (Sept. 2018), 35 pages. https://doi.org/10.1145/3233774
G. Ghinea and O. Ademoye. 2012. The sweet smell of success: Enhancing multimedia applications with olfaction. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 8, 1 (2012), 2.
G. Ghinea, F. Andres, and S. R. Gulliver. 2011. Multiple Sensorial Media Advances and Applications: New Developments in MulSeMedia: New Developments in MulSeMedia. Information Science Reference.
G. Ghinea, C. Timmerer, W. Lin, and S. R. Gulliver. 2014. Mulsemedia: State of the Art, Perspectives, and Challenges. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s, Article 17 (Oct. 2014), 23 pages. https://doi.org/10.1145/2617994
N. Murray, O. A. Ademoye, G. Ghinea, and G-M. Muntean. 2017. A Tutorial for Olfaction-Based Multisensorial Media Application Design and Evaluation. ACM Comp. Surveys 50, 5, Article 67 (2017), 30 pages. https://doi.org/10.1145/ 3108243
M. Obrist, C. Velasco, C. Vi, N. Ranasinghe, A. Israr, A. Cheok, C. Spence, and P. Gopalakrishnakone. 2016. Sensing the future of HCI: touch, taste, and smell user interfaces. interactions 23, 5 (2016), 40–49.
O. Petit, C. Velasco, and C. Spence. 2019. Digital Sensory Marketing: Integrating New Technologies Into Multisensory Online Experience. Journal of Interactive Marketing 45 (2019), 42 – 61. https://doi.org/10.1016/j.intmar.2018.07.004
N. Ranasinghe, P. Jain, N. Thi Ngoc Tram, K. C. R. Koh, D. Tolley, S. Karwita, L. Lien-Ya, Y. Liangkun, K. Shamaiah, C. Eason Wai Tung, C. C. Yen, and E. Y-L. Do. 2018. Season Traveller: Multisensory Narration for Enhancing the Virtual Reality Experience. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 577, 13 pages. https://doi.org/10.1145/3173574.3174151
E. B. Saleme, A. Covaci, G. Mesfn, C. A. S. Santos, and G. Ghinea. 2019. Mulsemedia DIY: A Survey of Devices and a Tutorial for Building Your Own Mulsemedia Environment. Comput. Surveys 52, 3, Article 58 (June 2019), 29 pages.
E. B. Saleme, C. A. S. Santos, R. A. Falbo, G. Ghinea, and F. Andres. 2019. MulseOnto: a Reference Ontology to Support the Design of Mulsemedia Systems. Journal of Universal Computer Science (2019), 26 pages.
E. B. Saleme, C. A. S. Santos, and G. Ghinea. 2019. Coping With the Challenges of Delivering Multiple Sensorial Media. IEEE MultiMedia 26, 2 (April 2019), 66–75. https://doi.org/10.1109/MMUL.2018.2873565
E. B. Saleme, C. A. S. Santos, and G. Ghinea. 2019. A mulsemedia framework for delivering sensory effects to heterogeneous systems. Multimedia Systems 25, 4 (01 Aug 2019), 421–447. https://doi.org/10.1007/s00530-019-00618-8
B. Serrano, R. M. Baños, and C. Botella. 2016. Virtual reality and stimulation of touch and smell for inducing relaxation: A randomized controlled trial. Computers in Human Behavior 55 (2016), 1–8.
B. S. Spencer. 2006. Incorporating the sense of smell into patient and haptic surgical simulators. IEEE Transactions on Information Technology in Biomedicine 10, 1 (Jan 2006), 168–173. https://doi.org/10.1109/TITB.2005.856851
R. J. Stone, R. Guest, P. Mahoney, D. Lamb, and C. Gibson. 2017. A ‘mixed reality’ simulator concept for future Medical Emergency Response Team training. Journal of the Royal Army Medical Corps 163, 4 (2017), 280–287. https: //doi.org/10.1136/jramc-2016-000726
Y. Sulema. 2016. Mulsemedia vs. Multimedia: State of the art and future trends. In 2016 International Conference on Systems, Signals and Image Processing (IWSSIP). 1–5. https://doi.org/10.1109/IWSSIP.2016.7502696
Irina Tal, Longhao Zou, Alexandra Covaci, Eva Ibarrola, Marilena Bratu, Gheorghita Ghinea, and Gabriel-Miro Muntean. 2019. Mulsemedia in Telecommunication and Networking Education: A Novel Teaching Approach that Improves the Learning Process. IEEE Communications Magazine 57, 11 (2019), 60–66.
Erik van der Meulen, Marina A Cidota, Stephan G Lukosch, Paulina JM Bank, Aadjan JC van der Helm, and Valentijn T Visch. 2016. A haptic serious augmented reality game for motor assessment of parkinson’s disease patients. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, 102–104.
C. Velasco and M. Obrist. 2021. Multisensory Experiences: A Primer. Frontiers in Computer Science 3 (2021), 12. https://doi.org/10.3389/fcomp.2021.614524
C. T. Vi, D. Arthur, and M. Obrist. 2018. TasteBud: Bring Taste Back into the Game. In Proceedings of the 3rd International Workshop on Multisensory Approaches to Human-Food Interaction (Boulder, CO, USA) (MHFI’18). ACM, New York, NY, USA, Article 1, 5 pages. https://doi.org/10.1145/3279954.3279955
B. Yecies. 2016. Transnational collaboration of the multisensory kind: exploiting Korean 4D cinema in China. Media International Australia 159, 1 (2016), 22–31.
L. Zou, I. Tal, A. Covaci, E. Ibarrola, G. Ghinea, and G-M. Muntean. 2017. Can Multisensorial Media Improve Learner Experience?. In Proceedings of the 8th ACM on Multimedia Systems Conference. ACM, 315–320.
Published
2021-06-21
How to Cite
COVACI, Alexandra; SALEME, Estêvão Bissoli; GHINEA, Gheorghita.
SensoryX ’21 Workshop on Multisensory Experiences at ACM IMX ’21 . In: WORKSHOP ON MULTISENSORY EXPERIENCES (SENSORYX), 1. , 2021, New York.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
DOI: https://doi.org/10.5753/sensoryx.2021.15682.
