skip to main content
10.1145/2820426.2820436acmotherconferencesArticle/Chapter ViewAbstractPublication PageswebmediaConference Proceedingsconference-collections
research-article

Specification of Multimodal Interactions in NCL

Published: 27 October 2015 Publication History

Abstract

This paper proposes an approach to integrate multimodal events--both user-generated, e.g., audio recognizer, motion sensors; and user-consumed, e.g., speech synthesizer, haptic synthesizer--into programming languages for the declarative specification of multimedia applications. More precisely, it presents extensions to the NCL (Nested Context Language) multimedia language. NCL is the standard declarative language for the development of interactive applications for Brazilian Digital TV and an ITU-T Recommendation for IPTV services. NCL applications extended with the multimodal features are presented as results. Historically, Human-Computer Interaction research community has been focusing on user-generated modalities, through studies on the user interaction. On the other hand, Multimedia community has been focusing on output modalities, through studies on timing and multimedia processing. The proposals in this paper is an attempt to integrate concepts of both research communities in a unique high-level programming framework, which aims to assist the authoring of multimedia/multimodal applications.

References

[1]
Batista, C.E.C.F. et al. 2010. Estendendo o uso das classes de dispositivos Ginga-NCL. WebMedia '10: Proceedings of the 16th Brazilian Symposium on Multimedia and the Web (2010).
[2]
Beckham, J.L. et al. 2001. Towards SMIL as a foundation for multimodal, multimedia applications. EUROSPEECH 2001 Scandinavia, 7th European Conference on Speech Communication and Technology (2001), 1363--1366.
[3]
Bulterman, D.C.A. and Rutledge, L.W. 2008. SMIL 3.0: Flexible Multimedia for Web, Mobile Devices and Daisy Talking Books. Springer Publishing Company, Incorporated.
[4]
Carvalho, L.A.M.C. et al. 2008. Architectures for Interactive Vocal Environment to Brazilian Digital TV Middleware. Proceedings of the 2008 Euro American Conference on Telematics and Information Systems (New York, NY, USA, 2008), 22:1--22:8.
[5]
Carvalho, L. and Macedo, H. 2010. Estendendo a NCL para Promover Interatividade Vocal em Aplicações Ginga na TVDi Brasileira. WebMedia '10: Proceedings of the 16th Brazilian Symposium on Multimedia and the Web (2010).
[6]
Costa, D. and Duarte, C. 2011. Adapting Multimodal Fission to User's Abilities. Universal Access in Human-Computer Interaction. Design for All and eInclusion. C. Stephanidis, ed. Springer Berlin Heidelberg. 347--356.
[7]
Coutaz, J. et al. 1995. Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. InterAct (1995), 115--120.
[8]
van Dam, A. 1997. Post-WIMP User Interfaces. Commun. ACM. 40, 2 (Feb. 1997), 63--67.
[9]
Dumas, B. et al. 2010. Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML. Journal on Multimodal User Interfaces. 3, 3 (Apr. 2010), 237--247.
[10]
Dumas, B. et al. 2009. Multimodal Interfaces: A Survey of Principles, Models and Frameworks. Human Machine Interaction. D. Lalanne and J. Kohlas, eds. Springer Berlin Heidelberg. 3--26.
[11]
Elouali, N. et al. 2013. Multimodal interaction: a survey from model driven engineering and mobile perspectives. Journal on Multimodal User Interfaces. 7, 4 (Dec. 2013), 351--370.
[12]
Lager, T. 2013. Statecharts and SCXML for Dialogue Management. Text, Speech, and Dialogue. I. Habernal and V. Matoušek, eds. Springer Berlin Heidelberg. 35--35.
[13]
Lucas, B. 2000. Voicexml. Communications of the ACM. 43, 9 (2000), 53.
[14]
Perez-Luque, M.J. and Little, T.D.C. 1996. A temporal reference framework for multimedia synchronization. IEEE Journal on Selected Areas in Communications. 14, 1 (Jan. 1996), 36--51.
[15]
Richard A. Bolt 1980. Put-That-There: Voice and Gesture at the Graphics Interface. Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques (New York, NY, USA, 1980), 262--270.
[16]
Soares, L.F.G. et al. 2009. Multiple Exhibition Devices in DTV Systems. Proceedings of the 17th ACM International Conference on Multimedia (New York, NY, USA, 2009), 281--290.
[17]
Soares, L.F.G. and Lima, G.F. NCL Handbook. Monografias em Ciência da Computação. 18/13.
[18]
Soares, L.F.G. and Rodrigues, R.F. 2005. Nested Context Model 3.0: Part 1 -- NCM Core (portuguese). Monografias em Ciência da Computação. 18/05 (2005).
[19]
Turk, M. 2014. Multimodal interaction: A review. Pattern Recognition Letters. 36, (Jan. 2014), 189--195.
[20]
Wang, K. 2002. SALT: a spoken language interface for web-based multimodal dialog systems. INTERSPEECH (2002).
[21]
XHTML+Voice Profile 1.0: 2001. http://www.w3.org/TR/xhtml+voice/. Accessed: 2015-05-11.

Cited By

View all
  • (2016)Extending NCL to Support Multiuser and Multimodal InteractionsProceedings of the 22nd Brazilian Symposium on Multimedia and the Web10.1145/2976796.2976869(39-46)Online publication date: 8-Nov-2016

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
WebMedia '15: Proceedings of the 21st Brazilian Symposium on Multimedia and the Web
October 2015
266 pages
ISBN:9781450339599
DOI:10.1145/2820426
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • CYTED: Ciência Y Tecnologia Para El Desarrollo
  • SBC: Brazilian Computer Society
  • FAPEAM: Fundacao de Amparo a Pesquisa do Estado do Amazonas
  • CNPq: Conselho Nacional de Desenvolvimento Cientifico e Tecn
  • CGIBR: Comite Gestor da Internet no Brazil
  • CAPES: Brazilian Higher Education Funding Council

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. mui
  2. multimodal interactions
  3. multimodal interfaces
  4. ncl
  5. nested context language

Qualifiers

  • Research-article

Conference

Webmedia '15
Sponsor:
  • CYTED
  • SBC
  • FAPEAM
  • CNPq
  • CGIBR
  • CAPES

Acceptance Rates

WebMedia '15 Paper Acceptance Rate 21 of 61 submissions, 34%;
Overall Acceptance Rate 270 of 873 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 30 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2016)Extending NCL to Support Multiuser and Multimodal InteractionsProceedings of the 22nd Brazilian Symposium on Multimedia and the Web10.1145/2976796.2976869(39-46)Online publication date: 8-Nov-2016

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media