Multimodal Annotation of ReINVenTA's Dataset

• Ana Carolina Loçasso Luz UFJF / CNPq http://orcid.org/0000-0002-1614-5138
• Gabrielly Braz UFJF https://orcid.org/0009-0006-2973-9046
• Lívia Pádua Ruiz UFJF / CNPq https://orcid.org/0009-0002-5289-1688
• Mariane de Carvalho Pinto UFJF https://orcid.org/0009-0005-4116-9931
• Frederico Belcavello UFJF http://orcid.org/0000-0001-5808-5201
• Natália Sathler Sigiliano UFJF https://orcid.org/0000-0002-8460-5546
• Tiago Torrent UFJF / CNPq https://orcid.org/0000-0001-5373-2297

DOI: https://doi.org/10.5753/stil.2023.233960

Abstract

This paper aims to present an application of the semantic-computacional model of FrameNet Brazil to the representation of multimodal objects. Therefore, it describes the steps involved in creating a subpart of the ReINVenTA Dataset, focusing on the semantic annotation of the TV series Pedro pelo Mundo for the modalities of text and dynamic images.

Keywords: Multimodal Annotation, FrameNet, Frame semantics, Computational model, Dataset

References

Baker, C. F., Fillmore, C. J. and Lowe, J. B. (1998). "The Berkeley FrameNet Project". In: COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics. https://doi.org/10.3115/980845.980860 https://aclanthology.org/P98-1013

Belcavello, F.; Viridiano, M.; Diniz Da Costa, A.; Matos, E. E.; Torrent, T. T. (2020). "Frame-Based Annotation of Multimodal Corpora: Tracking (A)Synchronies in Meaning Construction". In: Proceedings of the LREC International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet. Marseille, France: ELRA, p. 23-30. https://aclanthology.org/2020.framenet-1.4.pdf

Belcavello, F.; Viridiano, M.; Matos, E.; Torrent, T. T. (2022). "Charon: A FrameNet Annotation Tool for Multimodal Corpora". In: Proceedings of The 16th Lingusitic Annotation Workshop (LAW-XVI) within LREC2022. Marseille, France: ELRA, p. 91-96. https://doi.org/10.48550/arXiv.2205.11836 https://arxiv.org/abs/2205.11836

Belcavello, F. (2023). "FrameNet Annotation for Multimodal Corpora: devising a methodology for the semantic representation of text-image interactions in audiovisual productions". 135f. Tese (Doutorado em Linguística) — Faculdade de Letras, Universidade Federal de Juiz de Fora, Juiz de Fora. [link].

Dánnels, D.; Torrent, T. T.; Sigiliano, N. S.; Dobnik, S. (2022). "Beyond Strings of Characters: Resources meet NLP – Again". In: Volodina, E.; Dánnells, D.; Berdicevskis, A.; Forsberg, M.; Virk, S. (Org.). Live and Learn: Festschrift in honor of Lars Borin (pp. 29–36). Gothenburg: Institutionen för svenska, flerspråkighet och språkteknologi, Göteborgs Universitet. https://hdl.handle.net/2077/74254

Fillmore, C. J. (1982). "Frame semantics". In.: The linguistic society of Korea. Linguistics in the morning calm. Korea: Hanshin Publishing Company.

Fillmore, C. J.; Baker, C. (2009). "A Frames Approach To Semantic Analysis". In: Heine, B.; Narrog, H. (Orgs.). The Oxford Handbook Of Linguistic Analysis (pp. 313–340). Oxford: Oxford University Press. [link]. [link].

Steen, F., Hougaard, A., Joo, J., Olza, I., Cánovas, C., Pleshakova, A., Ray, S., Uhrig, P., Valenzuela, J., Woźny, J. and Turner, M. (2018) "Toward an infrastructure for data-driven multimodal communication research". Linguistics Vanguard, Vol. 4 (Issue 1), pp. 20170041. https://doi.org/10.1515/lingvan-2017-0041 [link].

Petruck, Miriam R. L. (1986) "Body Part Terminology in Hebrew: A Study in Lexical Semantics". Unpublished Ph.D. dissertation. University of California, Berkeley. https://escholarship.org/uc/item/7v7821mm

Salomão, M. M. M. (2009) "FrameNet Brasil: um trabalho em progresso". Calidoscópio, [S. l.], v. 7, n. 3, pp. 171–182. Disponível em: [link]. Acesso em: 6 ago. 2023. https://doi.org/10.4013/cld.2009.73.01.

Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermannz, R., & Zadeh, A. (2022). "Multimodal research in vision and language: A review of current and emerging trends". Information Fusion, 77, 149-171. https://doi.org/10.48550/arXiv.2010.09522 https://arxiv.org/abs/2010.09522