Mining Multimodal Patterns: Interpretable Video-Audio Relationships for Dataset Characterization and Semantic Retrieval
Resumo
The recent surge in large-scale multimodal datasets has advanced various video-audio tasks, from semantic retrieval to video-to-music generation; however, the explicit semantic rules governing how visual environments dictate musical styles remain poorly understood. This paper presents a generalizable, interpretable descriptive learning pipeline designed for multimodal dataset characterization, utilizing the generative GVMGEN benchmark as a primary case study. Because symbolic video representations suffer from inherent spatial sparsity, traditional mining algorithms succumb to sparsity bias, generating uninformative heuristics based on visual absences. To address this, our approach introduces a spatial relaxation layer that translates continuous visual embeddings into a discrete, binary Bag of Visual Words (BoVW), effectively eliminating temporal and positional constraints. We then combine global frequent pattern mining via FP-Growth with a positive-literal Constrained Beam Search to discover pure, discriminative visual signatures associated with diverse acoustic genres. The extracted semantic heuristics were validated in a downstream Semantic Video Retrieval task, where they operated as low-overhead Boolean filters, achieving a Retrieval Lift of up to 2.91× over blind search. Our findings provide an interpretable characterization of multimodal topologies, offering actionable prior knowledge that paves the way for robust multimedia indexing, general video-audio retrieval, and semantically-aware generative applications.
Palavras-chave:
Descriptive Learning, Multimodal Characterization, Subgroup Discovery, Semantic Retrieval, Video-to-Music
Referências
Rakesh Agrawal, Tomasz Imieliński, and Arun Swami. 1993. Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD international conference on Management of data. 207–216.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
Jiawei Han, Jian Pei, and Yiwen Yin. 2000. Mining frequent patterns without candidate generation. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (Dallas, Texas, USA) (SIGMOD ’00). Association for Computing Machinery, New York, NY, USA, 1–12. DOI: 10.1145/342009.335372
Francisco Herrera, Cristóbal José Carmona, Pedro González, and María José del Jesus. 2011. An overview on subgroup discovery: foundations and applications. Knowledge and Information Systems 29, 3 (2011), 495–525. DOI: 10.1007/s10115-010-0356-2
Haven Kim, Zachary Novack, Weihan Xu, Julian McAuley, and Hao-Wen Dong. 2025. Video-Guided Text-to-Music Generation Using Public Domain Movie Collections. arXiv:2506.12573 [cs.SD] [link]
Nada Lavrač, Peter Flach, and Blaz Zupan. 1999. Rule Evaluation Measures: A Unifying View. In Inductive Logic Programming, Sašo Džeroski and Peter Flach (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 174–185.
Nada Lavrač, Branko Kavšek, Peter Flach, and Ljupčo Todorovski. 2004. Subgroup Discovery with CN2-SD. Journal of Machine Learning Research 5 (2004), 153–188.
Florian Lemmerich and Martin Becker. 2018. pysubgroup: Easy-to-use subgroup discovery in python. In Joint European conference on machine learning and knowledge discovery in databases. Springer, 658–662.
Jiacheng Li, Haizhou Shi, Siliang Tang, Fei Wu, and Yueting Zhuang. 2019. Informative visual storytelling with cross-modal rules. In Proceedings of the 27th ACM International Conference on Multimedia. 2314–2322.
Ken-Hao Liu, Ming-Fang Weng, Chi-Yao Tseng, Yung-Yu Chuang, and Ming-Syan Chen. 2008. Association and temporal rule mining for post-filtering of semantic concept detection in video. IEEE Transactions on Multimedia 10, 2 (2008), 240–251.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing 508 (2022), 293–304.
Christopher D Manning. 2008. Introduction to information retrieval. Syngress Publishing,.
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matthew McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in Python. In Proceedings of the 14th Python in Science Conference.
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML).
Jorma Rissanen. 1978. Modeling by shortest data description. Automatica 14, 5 (1978), 465–471.
Timothy J Ross. 2005. Fuzzy logic with engineering applications. John Wiley & Sons.
Sivic and Zisserman. 2003. Video Google: A text retrieval approach to object matching in videos. In Proceedings ninth IEEE international conference on computer vision. IEEE, 1470–1477.
Jonatan Taminau, Ruben Hillewaere, Stijn Meganck, Darrell Conklin, Ann Nowé, and Bernard Manderick. 2009. Descriptive subgroup mining of folk music. In 2nd International Workshop on Machine Learning and Music (MML 2009), Bled, Slovenia.
Zeyue Tian, Zhaoyang Liu, Ruibin Yuan, Jiahao Pan, Qifeng Liu, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. 2025. VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 18782–18793.
V. Vijayakumar and R. Nedunchezhian. 2012. A study on video data mining. International Journal of Multimedia Information Retrieval 1, 3 (01 Oct 2012), 153–172. DOI: 10.1007/s13735-012-0016-2
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. 38–45.
Heda Zuo, Weitao You, Junxian Wu, Shihong Ren, Pei Chen, Mingxu Zhou, Yujia Lu, and Lingyun Sun. 2025. GVMGen: a general video-to-music generation model with hierarchical attentions. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’25/IAAI’25/EAAI’25). AAAI Press, Article 2576, 9 pages. DOI: 10.1609/aaai.v39i21.34474
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
Jiawei Han, Jian Pei, and Yiwen Yin. 2000. Mining frequent patterns without candidate generation. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (Dallas, Texas, USA) (SIGMOD ’00). Association for Computing Machinery, New York, NY, USA, 1–12. DOI: 10.1145/342009.335372
Francisco Herrera, Cristóbal José Carmona, Pedro González, and María José del Jesus. 2011. An overview on subgroup discovery: foundations and applications. Knowledge and Information Systems 29, 3 (2011), 495–525. DOI: 10.1007/s10115-010-0356-2
Haven Kim, Zachary Novack, Weihan Xu, Julian McAuley, and Hao-Wen Dong. 2025. Video-Guided Text-to-Music Generation Using Public Domain Movie Collections. arXiv:2506.12573 [cs.SD] [link]
Nada Lavrač, Peter Flach, and Blaz Zupan. 1999. Rule Evaluation Measures: A Unifying View. In Inductive Logic Programming, Sašo Džeroski and Peter Flach (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 174–185.
Nada Lavrač, Branko Kavšek, Peter Flach, and Ljupčo Todorovski. 2004. Subgroup Discovery with CN2-SD. Journal of Machine Learning Research 5 (2004), 153–188.
Florian Lemmerich and Martin Becker. 2018. pysubgroup: Easy-to-use subgroup discovery in python. In Joint European conference on machine learning and knowledge discovery in databases. Springer, 658–662.
Jiacheng Li, Haizhou Shi, Siliang Tang, Fei Wu, and Yueting Zhuang. 2019. Informative visual storytelling with cross-modal rules. In Proceedings of the 27th ACM International Conference on Multimedia. 2314–2322.
Ken-Hao Liu, Ming-Fang Weng, Chi-Yao Tseng, Yung-Yu Chuang, and Ming-Syan Chen. 2008. Association and temporal rule mining for post-filtering of semantic concept detection in video. IEEE Transactions on Multimedia 10, 2 (2008), 240–251.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing 508 (2022), 293–304.
Christopher D Manning. 2008. Introduction to information retrieval. Syngress Publishing,.
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matthew McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in Python. In Proceedings of the 14th Python in Science Conference.
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML).
Jorma Rissanen. 1978. Modeling by shortest data description. Automatica 14, 5 (1978), 465–471.
Timothy J Ross. 2005. Fuzzy logic with engineering applications. John Wiley & Sons.
Sivic and Zisserman. 2003. Video Google: A text retrieval approach to object matching in videos. In Proceedings ninth IEEE international conference on computer vision. IEEE, 1470–1477.
Jonatan Taminau, Ruben Hillewaere, Stijn Meganck, Darrell Conklin, Ann Nowé, and Bernard Manderick. 2009. Descriptive subgroup mining of folk music. In 2nd International Workshop on Machine Learning and Music (MML 2009), Bled, Slovenia.
Zeyue Tian, Zhaoyang Liu, Ruibin Yuan, Jiahao Pan, Qifeng Liu, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. 2025. VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 18782–18793.
V. Vijayakumar and R. Nedunchezhian. 2012. A study on video data mining. International Journal of Multimedia Information Retrieval 1, 3 (01 Oct 2012), 153–172. DOI: 10.1007/s13735-012-0016-2
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. 38–45.
Heda Zuo, Weitao You, Junxian Wu, Shihong Ren, Pei Chen, Mingxu Zhou, Yujia Lu, and Lingyun Sun. 2025. GVMGen: a general video-to-music generation model with hierarchical attentions. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’25/IAAI’25/EAAI’25). AAAI Press, Article 2576, 9 pages. DOI: 10.1609/aaai.v39i21.34474
Publicado
27/10/2026
Como Citar
OLIVEIRA, Franklin Sales de; MATOS, Diognei de; ONOFRE, Caio Henrique de Miranda.
Mining Multimodal Patterns: Interpretable Video-Audio Relationships for Dataset Characterization and Semantic Retrieval. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 32. , 2026, Lavras/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 319-327.
DOI: https://doi.org/10.5753/webmedia.2026.30927.
