Interactions in Multisensory Experiences: Toward a New Timeline Metaphor
Abstract
The PRIM project has the objective to allow everyone to produce multisensory interactions without programming and without the help of programmers. Taking inspiration from the simplicity of use of media editing software, our idea is to investigate how to enrich them to integrate interactions by modifying the editing timeline. After arguing that time and timeline are the major challenge to face before building a new software, this paper presents a new timeline metaphor, easy to understand, and that would allow to produce interactive mulsemedia and multisensory exercises once completed. Promising results show that people easily understand and accept this new timeline, which encourages us to continue in this way.
Keywords:
timeline metaphor, interactive mulsemedia, multisensory exercises, media editing software, music scores
References
C. Jost et al., "PRIM Project: Playing and Recording with Interactivity and Multisensoriality", in IMX 2021: ACM International Conference on Interactive Media Experiences, 2021.
C. Jost, B. Le Pevedic, O. El Barraj, and G. Uzan, "MulseBox: Portable multisensory interactive device," in Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics, 2019, vol. 2019-Octob, pp. 3956–3961.
M. Chetouani et al., "Cognitive services for elderly people: The ROBADOM project," 2010.
R. Casillas Figueroa, "Personal spaces for multisensory stimulation as support to rehabilitate patients with cognitive disabilities," in Proceedings of the XVIII International Conference on Human Computer Interaction - Interacción '17, 2017, pp. 1–2.
A. Covaci, L. Zou, I. Tal, G.-M. Muntean, and G. Ghinea, "Is Multimedia Multisensorial? - A Review of Mulsemedia Systems," ACM Comput. Surv., vol. 51, no. 5, pp. 1–35, 2018.
D. J. Malan, H. H. Leitner, D. J. Malan, and H. H. Leitner, "Scratch for budding computer scientists," in Proceedinds of the 38th SIGCSE technical symposium on Computer science education - SIGCSE '07, 2007, vol. 39, no. 1, p. 223.
E. Pot, J. Monceaux, R. Gelin, B. Maisonnier, and A. Robotics, "Choregraphe: A graphical tool for humanoid robot programming," Proc. - IEEE Int. Work. Robot Hum. Interact. Commun., pp. 46–51, 2009.
N. Fraser, "Ten things we’ve learned from Blockly," Proc. - 2015 IEEE Blocks Beyond Work. Blocks Beyond 2015, pp. 49–50, Dec. 2015.
G. Ghinea and O. Ademoye, "A user perspective of olfaction-enhanced mulsemedia," in Proceedings of the International Conference on Management of Emergent Digital EcoSystems - MEDES ’10, 2010, p. 277.
M. Waltl, B. Rainer, C. Timmerer, and H. Hellwagner, "An end-to-end tool chain for Sensory Experience based on MPEG-V," Signal Process. Image Commun., vol. 28, no. 2, pp. 136–150, Feb. 2013.
C. Timmerer, B. Rainer, and H. Hellwagner, "Sensory effect dataset and test setups Inter-Destination Multimedia Synchronization View project CONCERT View project," 2012.
M. Waltl, C. Timmerer, and H. Hellwagner, "A test-bed for quality of multimedia experience evaluation of Sensory Effects," in 2009 International Workshop on Quality of Multimedia Experience, 2009, pp. 145–150.
E. B. Saleme and C. A. S. Santos, "PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V," in Proceedings of the 21st Brazilian Symposium on Multimedia and the Web - WebMedia ’15, 2015, pp. 145–148.
D. P. De Mattos, C. Muchaluat-Saade, Débora, and G. Ghinea, "Beyond Multimedia Authoring: On the Need for Mulsemedia Authoring Tools," ACM Comput. Surv., vol. 54, no. 7, Jul. 2021.
D. P. De Mattos and D. C. Muchaluat-Saade, "Steve: A hypermedia authoring tool based on the simple interactive multimedia model," Proc. ACM Symp. Doc. Eng. 2018, DocEng 2018, Aug. 2018.
E. B. Saleme, C. A. S. Santos, and G. Ghinea, "Coping with the Challenges of Delivering Multiple Sensorial Media," IEEE Multimed., pp. 1–1, 2018.
E. Coronado, F. Mastrogiovanni, B. Indurkhya, and G. Venture, "Visual Programming Environments for End-User Development of intelligent and social robots, a systematic review," J. Comput. Lang., vol. 58, p. 100970, Jun. 2020.
K. Howland and J. Good, "Learning to communicate computationally with Flip: A bi-modal programming language for game creation," Comput. Educ., vol. 80, pp. 224–240, Jan. 2015.
J. Brooke, "System usability scale (SUS): a quick-and-dirty method of system evaluation user information," Reading, UK Digit. Equip. co ltd, vol. 43, pp. 1–7, 1986.
C. Jost, B. Le Pevedic, O. El Barraj, and G. Uzan, "MulseBox: Portable multisensory interactive device," in Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics, 2019, vol. 2019-Octob, pp. 3956–3961.
M. Chetouani et al., "Cognitive services for elderly people: The ROBADOM project," 2010.
R. Casillas Figueroa, "Personal spaces for multisensory stimulation as support to rehabilitate patients with cognitive disabilities," in Proceedings of the XVIII International Conference on Human Computer Interaction - Interacción '17, 2017, pp. 1–2.
A. Covaci, L. Zou, I. Tal, G.-M. Muntean, and G. Ghinea, "Is Multimedia Multisensorial? - A Review of Mulsemedia Systems," ACM Comput. Surv., vol. 51, no. 5, pp. 1–35, 2018.
D. J. Malan, H. H. Leitner, D. J. Malan, and H. H. Leitner, "Scratch for budding computer scientists," in Proceedinds of the 38th SIGCSE technical symposium on Computer science education - SIGCSE '07, 2007, vol. 39, no. 1, p. 223.
E. Pot, J. Monceaux, R. Gelin, B. Maisonnier, and A. Robotics, "Choregraphe: A graphical tool for humanoid robot programming," Proc. - IEEE Int. Work. Robot Hum. Interact. Commun., pp. 46–51, 2009.
N. Fraser, "Ten things we’ve learned from Blockly," Proc. - 2015 IEEE Blocks Beyond Work. Blocks Beyond 2015, pp. 49–50, Dec. 2015.
G. Ghinea and O. Ademoye, "A user perspective of olfaction-enhanced mulsemedia," in Proceedings of the International Conference on Management of Emergent Digital EcoSystems - MEDES ’10, 2010, p. 277.
M. Waltl, B. Rainer, C. Timmerer, and H. Hellwagner, "An end-to-end tool chain for Sensory Experience based on MPEG-V," Signal Process. Image Commun., vol. 28, no. 2, pp. 136–150, Feb. 2013.
C. Timmerer, B. Rainer, and H. Hellwagner, "Sensory effect dataset and test setups Inter-Destination Multimedia Synchronization View project CONCERT View project," 2012.
M. Waltl, C. Timmerer, and H. Hellwagner, "A test-bed for quality of multimedia experience evaluation of Sensory Effects," in 2009 International Workshop on Quality of Multimedia Experience, 2009, pp. 145–150.
E. B. Saleme and C. A. S. Santos, "PlaySEM: a Platform for Rendering MulSeMedia Compatible with MPEG-V," in Proceedings of the 21st Brazilian Symposium on Multimedia and the Web - WebMedia ’15, 2015, pp. 145–148.
D. P. De Mattos, C. Muchaluat-Saade, Débora, and G. Ghinea, "Beyond Multimedia Authoring: On the Need for Mulsemedia Authoring Tools," ACM Comput. Surv., vol. 54, no. 7, Jul. 2021.
D. P. De Mattos and D. C. Muchaluat-Saade, "Steve: A hypermedia authoring tool based on the simple interactive multimedia model," Proc. ACM Symp. Doc. Eng. 2018, DocEng 2018, Aug. 2018.
E. B. Saleme, C. A. S. Santos, and G. Ghinea, "Coping with the Challenges of Delivering Multiple Sensorial Media," IEEE Multimed., pp. 1–1, 2018.
E. Coronado, F. Mastrogiovanni, B. Indurkhya, and G. Venture, "Visual Programming Environments for End-User Development of intelligent and social robots, a systematic review," J. Comput. Lang., vol. 58, p. 100970, Jun. 2020.
K. Howland and J. Good, "Learning to communicate computationally with Flip: A bi-modal programming language for game creation," Comput. Educ., vol. 80, pp. 224–240, Jan. 2015.
J. Brooke, "System usability scale (SUS): a quick-and-dirty method of system evaluation user information," Reading, UK Digit. Equip. co ltd, vol. 43, pp. 1–7, 1986.
Published
2022-06-22
How to Cite
JOST, Céline; LE PÉVÉDIC, Brigitte; DEBLOOS, Justin; UZAN, Gérard.
Interactions in Multisensory Experiences: Toward a New Timeline Metaphor. In: WORKSHOP ON MULTISENSORY EXPERIENCES (SENSORYX), 2. , 2022, Aveiro.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
DOI: https://doi.org/10.5753/sensoryx.2022.20003.
