Segmentação de vídeo em cenas - uma abordagem multimodal baseada em Transformers

• Marcus Vinicius Cerrato USP
• Wan Song Rocha USP
• Rudinei Goularte USP

DOI: https://doi.org/10.5753/webmedia.2026.29751

Resumo

Automatic temporal video scene segmentation remains a challenging and open problem in the Multimedia field. While recent state-of-the-art approaches shift toward multimodal paradigms, conventional methods often rely on rigid local fusion strategies that may fail to capture long-range temporal dependencies or handle cross-modal asynchronous perturbations. To alleviate these limitations, this paper proposes a novel multimodal Transformer-based architecture configured with a joint space-temporal self-attention mechanism. The proposed method introduces an adaptive fusion strategy that integrates visual and acoustic streams, represented by dense MFCC embeddings, at the shot level and dynamically regulated by a gating mechanism. Evaluated under a cross-validation protocol on a public dataset, the proposed approach exceeds the average scores from related work.

Palavras-chave: Multimedia, Video, Transformers, Temporal Scene Segmentation, Scene Boundary Detection

Referências

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 24206–24221. [link]

John Edison Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A. González. 2017. Gated Multimodal Units for Information Fusion. In International Conference on Learning Representations.

Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021. ViViT: A Video Vision Transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 6836–6846.

Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. 2015. A Deep Siamese Network for Scene Detection in Broadcast Videos. In Proceedings of the 23rd ACM International Conference on Multimedia (Brisbane, Australia) (MM ’15). Association for Computing Machinery, New York, NY, USA, 1199–1202. DOI: 10.1145/2733373.2806316

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is Space-Time Attention All You Need for Video Understanding?. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 813–824. [link]

Manfred Del Fabro and Laszlo Böszörmenyi. 2013. State-of-the-art and future challenges in video scene detection: a survey. Multimedia Systems 19, 5 (oct 2013), 427–454. DOI: 10.1007/s00530-013-0306-4

Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18, 5-6 (2005), 602–610. DOI: 10.1016/j.neunet.2005.06.042

Bo Han and Weiguo Wu. 2011. Video scene segmentation using a novel boundary evaluation criterion and dynamic programming. In 2011 IEEE International Conference on Multimedia and Expo. 1–6. DOI: 10.1109/ICME.2011.6012001

Rodrigo Mitsuo Kishi, Tiago Henrique Trojahn, and Rudinei Goularte. 2019. Correlation based feature fusion for the temporal video scene segmentation task. Multimedia Tools and Applications 78, 11 (01 Jun 2019), 15623–15646. DOI: 10.1007/s11042-018-6959-4

Tsung-Yi Lin, Serge Maire, Michael Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740–755.

Siva Jyothi Lingam and Sudha Pelluri. 2025. Multi-modal Video Summarization: A Comprehensive Survey. In 2025 International Conference on Robotics and Mechatronics (ICRM). 1–6. DOI: 10.1109/ICRM66809.2025.11349049

Dong Liu, Nagendra Kamath, Subhabrata Bhattacharya, and Rohit Puri. 2020. Adaptive context reading network for movie scene detection. IEEE Transactions on Circuits and Systems for Video Technology 31, 9 (2020), 3559–3574.

Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2022. Video Swin Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3202–3211.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. [link]

Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin. 2020. A Local-to-Global Approach to Multi-Modal Movie Scene Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10143–10152. DOI: 10.1109/CVPR42600.2020.01016

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A Cleaned, Hypernymed, Image Alttext Dataset For Automatic Image Captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2556–2565. DOI: 10.18653/v1/P18-1238

A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. 2000. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 12 (2000), 1349–1380. DOI: 10.1109/34.895972

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. VideoBERT: A Joint Model for Video and Language Representation Learning. In Proceedings of the IEEE/ CVF International Conference on Computer Vision (ICCV).

Tiago H. Trojahn, Rodirgo M. Kishi, and Rudinei Goularte. 2018. A new multimodal deep-learning model to video scene segmentation. In Proceedings of the 24th Brazilian Symposium on Multimedia and the Web (Salvador, BA, Brazil) (WebMedia ’18). ACM, New York, NY, USA, 205–212. DOI: 10.1145/3243082.3243108

Shu-Ming Tseng, Zhi-Ting Yeh, Chia-Yang Wu, Jia-Bin Chang, and Mehdi Norouzi. 2023. Video Scene Detection Using Transformer Encoding Linker Network (TELNet). Sensors 23, 16 (2023). DOI: 10.3390/s23167050

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. 5998–6008. [link]

Jeroen Vendrig and Marcel Worring. 2002. Systematic evaluation of logical story unit segmentation. IEEE Transactions on Multimedia 4, 4 (Dec. 2002), 492–499. DOI: 10.1109/TMM.2002.802021

Peng Xu, Xiatian Zhu, and David A. Clifton. 2023. Multimodal Learning With Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 10 (2023), 12113–12132. DOI: 10.1109/TPAMI.2023.3275156

Tian Ye, Yang Liu, Mengyu Yang, LanShan Zhang, and Zhigang Li. 2024. Semantic Fusion Based Graph Network for Video Scene Detection. In 2024 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–8.

Luowei Zhou, Changxu Xu, and Jason J Corso. 2018. Towards Automatic Learning of Procedures from Web Instructional Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7054–7063.

Tianfei Zhou, Fatih Porikli, David J. Crandall, Luc Van Gool, and Wenguan Wang. 2023. A Survey on Deep Learning Technique for Video Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 6 (2023), 7099–7122. DOI: 10.1109/TPAMI.2022.3225573