SONA: Geração Automatizada de Audiodescrição com Ajuste Temporal a Lacunas de Fala

• Pedro Alves Valentim UFF
• Débora C. Muchaluat-Saade UFF

DOI: https://doi.org/10.5753/webmedia.2026.30922

Resumo

Audio description (AD) is a critical accessibility resource for blind and low-vision audiences, but producing AD scripts remains time-intensive and costly when performed manually. This paper introduces SONA (System for Optimized Narration and Audiodescription), a hybrid, end-to-end pipeline that generates AD assets directly from a final audiovisual media file with minimal user input. SONA segments a video into shots, extracts and clusters faces for character identification, tracks characters through color-based visual annotations, and combines environment summarization with structured inference of actions, interactions, facial expressions, and salient objects using multimodal large language models. The resulting per-shot narrations are orchestrated into speech-free intervals detected by voice activity detection, producing a timestamped WebVTT AD script and a synthesized AD track with configurable output language and voice. A core contribution is an explicit fit-to-gap mechanism that accepts a narration segment only when its measured synthesized duration respects the assigned silence gap, using bounded time compression and iterative LLM-based shortening when needed. We evaluate SONA on MPII-MD, TV-AD, and a full-length feature film case study, complement the analysis with component ablations, and report an expert qualitative assessment of two complete film outputs. Results indicate low lexical overlap but stronger semantic agreement, a clear timing benefit from fit-to-gap enforcement, and practical usefulness as a drafting tool for AD workflows.

Palavras-chave: Audiodescrição, acessibilidade, compreensão de vídeo, modelos de linguagem, alinhamento temporal, síntese de voz

Referências

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics, 65–72. [link]

G. Bradski. 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000).

Virginia Pinto Campos, Tiago Maritan U de Araújo, and Guido L de Souza Filho. 2014. CineAD: Um Sistema de Geração Automática de Roteiros de Audiodescrição. In Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia). SBC, 1–8.

Brandon Castellano. 2020. Pyscenedetect. Last accessed (2020).

Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. 2019. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR.

FFmpeg Developers. 2025. ffmpeg tool. [link]. Version 8.0 “Huffman” [Software].

Hexgrad. 2025. Kokoro-82M: Open-weight text-to-speech model. [link]. DOI: 10.57967/hf/4329, Apache-2.0 License.

James Lakritz and Andrew Salway. 2006. The semi-automatic generation of audio description from screenplays. Dept. of Computing Technical Report CS-06-05, University of Surrey (2006).

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004). Association for Computational Linguistics, 25–26. [link]

Leland McInnes, John Healy, Steve Astels, et al. 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw. 2, 11 (2017), 205.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 311–318. DOI: 10.3115/1073083.1073135

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 [cs.CV] [link]

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 3982–3992. DOI: 10.18653/v1/D19-1410

James Robert, Marc Webbie, et al. 2018. Pydub. [link]

Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015. A Dataset for Movie Description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3202–3212. DOI: 10.1109/CVPR.2015.7298940

Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11987–11997.

Ollama Team. 2025. Ollama. [link]. Version 0.13.0.

Silero Team. 2024. Silero VAD: pre-trained enterprise-grade Voice Activity Detector (VAD), Number Detector and Language Classifier. [link].

Guido Van Rossum and Fred L. Drake. 2009. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA.

W3C. 2019. WebVTT: The Web Video Text Tracks Format. Technical Report W3C Recommendation. World Wide Web Consortium. [link]

Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Eshika Khandelwal, Gül Varol, Weidi Xie, and Andrew Zisserman. 2025. Shotby-Shot: Film-Grammar-Aware Training-Free Audio Description Generation. arXiv preprint arXiv:2504.01020 (2025).

Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisserman. 2024. Autoad-zero: A trainingfree framework for zero-shot audio description. In Proceedings of the Asian Conference on Computer Vision. 2265–2281.

Xiaojun Ye, Junhao Chen, Xiang Li, Haidong Xin, Chao Li, Sheng Zhou, and Jiajun Bu. 2024. MMAD: Multi-modal movie audio description. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 11415–11428.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In Proceedings of the International Conference on Learning Representations (ICLR). OpenReview.net. [link]