Understanding genre similarity in Brazilian music through Vision Transformer embeddings
Resumo
This work investigates how musical genre similarity is represented in the latent space learned by a Vision Transformer (ViT) model fine-tuned on Brazilian regional music. We present BYRM, a curated dataset that contains 1,082 tracks across ten culturally diverse genres. Our analysis focuses on the best-performing configuration identified in previous experiments, which uses 10-second segments extracted from the 90 to 120 second excerpt of each track. Mel-spectrograms were used as input to train the ViT, from which time-local embeddings were extracted. Dimensionality reduction techniques (PCA, t-SNE, and UMAP) were applied to visualize the latent space, and cosine similarity was computed to assess inter-genre proximity. The model achieved 81.94% accuracy and 81.84% F1-score under the best configuration, demonstrating its ability to learn discriminative representations. The similarity analysis shows that the ViT effectively captures stylistic relationships between related genres, such as samba and pagode or vaneira and xote ga´ucho, while maintaining separation from more distinct styles like Brazilian rock. These findings provide new insights into genre similarity modeling using transformer-based embeddings in a culturally rich and musically complex context.Referências
Jean-Julien Aucouturier and Elias Pampalk. Introduction–from genres to tags: A little epistemology of music information retrieval research. Journal of New Music Research, 37(2):87–92, 2008.
Carlos N Silla Jr, Celso AA Kaestner, and Alessandro L Koerich. Automatic music genre classification using ensemble of classifiers. In 2007 IEEE International Conference on Systems, Man and Cybernetics, pages 1687–1692. IEEE, 2007.
Bob L Sturm. The state of the art ten years after a state of the art: Future research in music information retrieval. Journal of new music research, 43(2):147–172, 2014.
Raul de Araújo Lima, Rômulo César Costa de Sousa, Hélio Lopes, and Simone Diniz Junqueira Barbosa. Brazilian lyrics-based music genre classification using a blstm network. In International Conference on Artificial Intelligence and Soft Computing, pages 525–534. Springer, 2020.
Szymon Muszynski and Zbigniew Tarapata. Methods of automated music comparison based on multi-objective metrics of network similarity. Applied Sciences, 13(6):3567, 2023.
Lucas Cassano, Elisa de Oliveira Morais Nacur Cassano, and Robert Moses Pechman. Passado e presente da música carioca: Narrativas urbanas e evolução tecnológica. DEBATES-Cadernos do Programa de Pós-Graduação em Música, 28:e282408–e282408, 2024.
Adriana Fernandes. Music, migrancy, and modernity: a study of Brazilian forró. University of Illinois at Urbana-Champaign, 2005.
Marcio Guedes Correa. O conceito de gênero musical no repertório e nas áreas de antropologia, comunicação, etnomusicologia e musicologia. ARJ–Art Research Journal: Revista de Pesquisa em Artes, 5(2), 2018.
Armando Alexandre Castro. Axé music: mitos, verdades e world music. Per Musi, pages 203–217, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Timothy P Byron, Carl T Rushworth, and Maxwell J Stewart. Popular music excerpts are rated as more memorable and salient if they involve vocals, compound hooks, and choruses. Music Perception: An Interdisciplinary Journal, 42(3):197–206, 2025.
Victória de Souza Guimarães and Rosiane de Freitas. Byrm: Brazilian youtube regional music dataset, 2025.
Xin Zhang and Zbigniew W Ras. Analysis of sound features for music timbre recognition. In 2007 International Conference on Multimedia and Ubiquitous Engineering (MUE’07), pages 3–8. IEEE, 2007.
Muhammad Turab, Teerath Kumar, Malika Bendechache, and Takfarinas Saber. Investigating multi-feature selection and ensembling for audio classification. arXiv preprint arXiv:2206.07511, 2022.
Jan Schlüter and Thomas Grill. Exploring data augmentation for improved singing voice detection with neural networks. In ISMIR, pages 121–126, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37–52, 1987.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
Janne Spijkervet and John Ashley Burgoyne. Contrastive learning of musical representations. arXiv preprint arXiv:2103.09410, 2021.
Julia Barnett, Hugo Flores Garcia, and Bryan Pardo. Exploring musical roots: Applying audio embeddings to empower influence attribution for a generative music model. arXiv preprint arXiv:2401.14542, 2024.
Mads Krogh. Music-genre abstraction, dissemination and trajectories. Dansk Musikforskning Online, 9:83–108, 2019.
Cameron Armijo and Pablo Rivas. Exploring visual embedding spaces induced by vision transformers for online auto parts marketplaces. arXiv preprint arXiv:2502.05756, 2025.
Ori Yonay, Tracy Hammond, and Tianbao Yang. Myna: Masking-based contrastive learning of musical representations. arXiv preprint arXiv:2502.12511, 2025.
Mohamadreza Sheikh Fathollahi and Farbod Razzazi. Music similarity measurement and recommendation system using convolutional neural networks. International Journal of Multimedia Information Retrieval, 10:43–53, 2021.
Hongzhi Shu, Xinglin Li, Hongyu Jiang, Minghao Fu, and Xinyu Li. Benchmarking sub-genre classification for mainstage dance music. arXiv preprint arXiv:2409.06690, 2024.
Xuejiao Luo, Vera Hoveling, and Elmar Eisemann. Musicon: Glyph-based design for music visualization and retrieval. 2024.
Jefferson Martins De Sousa, Eanes Torres Pereira, and Luciana Ribeiro Veloso. A robust music genre classification approach for global and regional music datasets evaluation. In 2016 IEEE international conference on digital signal processing (DSP), pages 109–113. IEEE, 2016.
Júlia Luiza Conceição, Rosiane de Freitas, Bruno Gadelha, João Gustavo Kienen, Sérgio Anders, and Brendo Cavalcante. Applying supervised learning techniques to brazilian music genre classification. In 2020 XLVI Latin American Computing Conference (CLEI), pages 102–107. IEEE, 2020.
Diego Furtado Silva, Micael Valterlânio da Silva, Ricardo Szram Filho, and Angelo Cesar Mendes da Silva. On the fusion of multiple audio representations for music genre classification. In Simpósio Brasileiro de Computação Musical (SBCM), pages 37–44. SBC, 2021.
Diego Furtado Silva, Angelo Cesar Mendes da Silva, Luís Felipe Ortolan, and Ricardo Marcondes Marcacini. On generalist and domain-specific music classification models and their impacts on brazilian music genre recognition. In Simpósio Brasileiro de Computação Musical (SBCM), pages 60–67. SBC, 2021.
Angelo Cesar Mendes da Silva, Diego Furtado Silva, and Ricardo Marcondes Marcacini. 4mula: A multitask, multimodal, and multilingual dataset of music lyrics and audio features. In Proceedings of the Brazilian Symposium on Multimedia and the Web, pages 145–148, 2020.
V. Guimarães, J. G. Kienen, and R. de Freitas. Segment-based evaluation of music genre classification models with the byrm dataset. In Proceedings of the Symposium on Knowledge Discovery, Mining and Learning (KDMiLe), 2025. To appear.
Carlos N Silla Jr, Celso AA Kaestner, and Alessandro L Koerich. Automatic music genre classification using ensemble of classifiers. In 2007 IEEE International Conference on Systems, Man and Cybernetics, pages 1687–1692. IEEE, 2007.
Bob L Sturm. The state of the art ten years after a state of the art: Future research in music information retrieval. Journal of new music research, 43(2):147–172, 2014.
Raul de Araújo Lima, Rômulo César Costa de Sousa, Hélio Lopes, and Simone Diniz Junqueira Barbosa. Brazilian lyrics-based music genre classification using a blstm network. In International Conference on Artificial Intelligence and Soft Computing, pages 525–534. Springer, 2020.
Szymon Muszynski and Zbigniew Tarapata. Methods of automated music comparison based on multi-objective metrics of network similarity. Applied Sciences, 13(6):3567, 2023.
Lucas Cassano, Elisa de Oliveira Morais Nacur Cassano, and Robert Moses Pechman. Passado e presente da música carioca: Narrativas urbanas e evolução tecnológica. DEBATES-Cadernos do Programa de Pós-Graduação em Música, 28:e282408–e282408, 2024.
Adriana Fernandes. Music, migrancy, and modernity: a study of Brazilian forró. University of Illinois at Urbana-Champaign, 2005.
Marcio Guedes Correa. O conceito de gênero musical no repertório e nas áreas de antropologia, comunicação, etnomusicologia e musicologia. ARJ–Art Research Journal: Revista de Pesquisa em Artes, 5(2), 2018.
Armando Alexandre Castro. Axé music: mitos, verdades e world music. Per Musi, pages 203–217, 2010.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Timothy P Byron, Carl T Rushworth, and Maxwell J Stewart. Popular music excerpts are rated as more memorable and salient if they involve vocals, compound hooks, and choruses. Music Perception: An Interdisciplinary Journal, 42(3):197–206, 2025.
Victória de Souza Guimarães and Rosiane de Freitas. Byrm: Brazilian youtube regional music dataset, 2025.
Xin Zhang and Zbigniew W Ras. Analysis of sound features for music timbre recognition. In 2007 International Conference on Multimedia and Ubiquitous Engineering (MUE’07), pages 3–8. IEEE, 2007.
Muhammad Turab, Teerath Kumar, Malika Bendechache, and Takfarinas Saber. Investigating multi-feature selection and ensembling for audio classification. arXiv preprint arXiv:2206.07511, 2022.
Jan Schlüter and Thomas Grill. Exploring data augmentation for improved singing voice detection with neural networks. In ISMIR, pages 121–126, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37–52, 1987.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
Janne Spijkervet and John Ashley Burgoyne. Contrastive learning of musical representations. arXiv preprint arXiv:2103.09410, 2021.
Julia Barnett, Hugo Flores Garcia, and Bryan Pardo. Exploring musical roots: Applying audio embeddings to empower influence attribution for a generative music model. arXiv preprint arXiv:2401.14542, 2024.
Mads Krogh. Music-genre abstraction, dissemination and trajectories. Dansk Musikforskning Online, 9:83–108, 2019.
Cameron Armijo and Pablo Rivas. Exploring visual embedding spaces induced by vision transformers for online auto parts marketplaces. arXiv preprint arXiv:2502.05756, 2025.
Ori Yonay, Tracy Hammond, and Tianbao Yang. Myna: Masking-based contrastive learning of musical representations. arXiv preprint arXiv:2502.12511, 2025.
Mohamadreza Sheikh Fathollahi and Farbod Razzazi. Music similarity measurement and recommendation system using convolutional neural networks. International Journal of Multimedia Information Retrieval, 10:43–53, 2021.
Hongzhi Shu, Xinglin Li, Hongyu Jiang, Minghao Fu, and Xinyu Li. Benchmarking sub-genre classification for mainstage dance music. arXiv preprint arXiv:2409.06690, 2024.
Xuejiao Luo, Vera Hoveling, and Elmar Eisemann. Musicon: Glyph-based design for music visualization and retrieval. 2024.
Jefferson Martins De Sousa, Eanes Torres Pereira, and Luciana Ribeiro Veloso. A robust music genre classification approach for global and regional music datasets evaluation. In 2016 IEEE international conference on digital signal processing (DSP), pages 109–113. IEEE, 2016.
Júlia Luiza Conceição, Rosiane de Freitas, Bruno Gadelha, João Gustavo Kienen, Sérgio Anders, and Brendo Cavalcante. Applying supervised learning techniques to brazilian music genre classification. In 2020 XLVI Latin American Computing Conference (CLEI), pages 102–107. IEEE, 2020.
Diego Furtado Silva, Micael Valterlânio da Silva, Ricardo Szram Filho, and Angelo Cesar Mendes da Silva. On the fusion of multiple audio representations for music genre classification. In Simpósio Brasileiro de Computação Musical (SBCM), pages 37–44. SBC, 2021.
Diego Furtado Silva, Angelo Cesar Mendes da Silva, Luís Felipe Ortolan, and Ricardo Marcondes Marcacini. On generalist and domain-specific music classification models and their impacts on brazilian music genre recognition. In Simpósio Brasileiro de Computação Musical (SBCM), pages 60–67. SBC, 2021.
Angelo Cesar Mendes da Silva, Diego Furtado Silva, and Ricardo Marcondes Marcacini. 4mula: A multitask, multimodal, and multilingual dataset of music lyrics and audio features. In Proceedings of the Brazilian Symposium on Multimedia and the Web, pages 145–148, 2020.
V. Guimarães, J. G. Kienen, and R. de Freitas. Segment-based evaluation of music genre classification models with the byrm dataset. In Proceedings of the Symposium on Knowledge Discovery, Mining and Learning (KDMiLe), 2025. To appear.
Publicado
15/09/2025
Como Citar
GUIMARÃES, Victória; KIENEN, João Gustavo; DE FREITAS, Rosiane.
Understanding genre similarity in Brazilian music through Vision Transformer embeddings. In: SIMPÓSIO BRASILEIRO DE COMPUTAÇÃO MUSICAL (SBCM), 19. , 2025, Campinas/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 203-210.
DOI: https://doi.org/10.5753/sbcm.2025.13212.
