Visual Foundation Model-Based Classification of Characters in Narrative Media - Cunha, Yan Martins B. Gurevitz; Moraes, Daniel de Sousa; Busson, Antonio José Grandson; Duarte, Julio Cesar; Colcher, Sérgio (Proceedings of the Brazilian Symposium on Multimedia and the Web (WebMedia), 2025)
The field of narratology has long worked on different ways to classify characters according to their narrative role and importance; meanwhile, studies on character design have observed the existence of unclear patterns that correlate specific visual features and those narrative classifications. Therefore, it becomes worth considering if modern image classification methods can be used to not only achieve satisfactory performance on this task, but also help us learn about the patterns that shape it. The task of classifying characters carries a vast array of practical uses, such as creating recommender systems based on specific visual-narrative correlations, automatic annotation for media preservation, adaptive interactive storytelling based on how a character is perceived, and the creation of education tools on how those patterns change depending on culture. In this work, we construct a dataset customized to assessing both performance and visual–narrative correlations within a specific medium, conduct an anonymous survey to evaluate human performance and tendencies, and compare the results with those of a traditional CNN classifier and a Foundation Model–augmented approach. Both models outperform the survey participants, with the latter achieving 76.93% accuracy, 18.11% higher than human performance, while also providing insights into the visual cues most closely tied to the given narrative roles.