Arquitetura para Geração de Audiodescrição Usando VLMs e LLMs

• Pedro Alves Valentim UFF
• Débora C. Muchaluat-Saade UFF

DOI: https://doi.org/10.5753/webmedia_estendido.2026.35032

Resumo

Audio description is an accessibility resource that conveys visual information through an additional narration track, but its manual production remains costly and time-consuming. This dissertation proposes a modular architecture for automatically generating audio-description scripts and synthesized tracks directly from final audiovisual media. The architecture integrates temporal segmentation, visual-content analysis, multimodal language models, text generation, and synchronization with the original soundtrack. A proof of concept, named SONA, was implemented to assess the proposed architecture. Experiments with public audio-description benchmarks and feature films indicate stronger semantic than lexical agreement with professional references. An expert assessment also indicates good intelligibility and practical value of the generated scripts as editable drafts. The results support the proposed architecture as a foundation for human-supervised audio-description authoring systems.

Palavras-chave: audio description, accessibility, multimedia systems, multimodal language models, video understanding, temporal alignment

Referências

Virginia Pinto Campos, Tiago Maritan U. de Araújo, and Guido L. de Souza Filho. 2014. CineAD: Um Sistema de Geração Automática de Roteiros de Audiodescrição. In Simpósio Brasileiro de Sistemas Multimídia e Web. SBC, 1–8.

James Lakritz and Andrew Salway. 2006. The Semi-Automatic Generation of Audio Description from Screenplays. Technical Report CS-06-05. University of Surrey.

Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015. A Dataset for Movie Description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3202–3212.

Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisserman. 2024. AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description. In Proceedings of the Asian Conference on Computer Vision. 2265–2281.