Low-Complexity Sign Language Recognition: A Cross-Modal Distillation Approach for State-Space Models

• Victor Costa UFPel
• Lucas Superti UFPel
• Brenda Salenave Santana UFPel
• Tatiana Bolivar Lebedeff UFPel
• Guilherme Corrêa UFPel

DOI: https://doi.org/10.5753/webmedia.2026.30763

Resumo

Sign Language Recognition (SLR) is a fundamental challenge for digital accessibility and interactive multimedia systems. While state-of-the-art approaches that rely on 3D Convolutional Neural Networks (e.g., I3D) achieve high accuracy by extracting dense spatiotemporal features directly from RGB videos, their high computational cost makes them prohibitive for real-time edge computing applications. Conversely, lightweight models based on spatial landmarks significantly reduce computational overhead but suffer from a severe drop in accuracy due to the loss of implicit visual semantic context, such as subtle hand depths and environmental textures. To address the dichotomy between computational efficiency and semantic richness, this paper proposes a Cross-Modal Knowledge Distillation pipeline for Brazilian Sign Language (LIBRAS) recognition. We utilize an I3D network operating on raw video as the Teacher model to transfer "soft probabilities" to a Mamba 1D Student model operating exclusively on spatial coordinates extracted via MediaPipe. Experimental results demonstrate that our cross-modal distillation approach successfully bridges the semantic gap. While a vanilla Mamba 1D model (trained without distillation) achieves only 71% signer-independent accuracy, our proposed distilled approach elevates this performance to 78%. Remarkably, this 7% absolute improvement introduces zero computational overhead during inference. The proposed distilled Mamba 1D model achieves real-time latency (0.21 ms), requires only 9.81 MB of VRAM, and consumes a mere 0.004 Joules per inference, representing a 99.7% reduction in energy consumption compared to the video-based baseline. These results demonstrate the viability of deploying highly accurate, semantically aware sign language translation systems natively on resource-constrained mobile and IoT devices.

Palavras-chave: LIBRAS, sign language, sign language recognition, neural networks, knowledge distillation, Mamba

Referências

Carlos Eduardo G. R. Alves, Francisco De A. Boldt, and Thiago M. Paixão. 2024. Enhancing Brazilian Sign Language Recognition Through Skeleton Image Representation. In 2024 37th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). 1–6. DOI: 10.1109/SIBGRAPI62404.2024.10716301

Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2021. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields . IEEE Transactions on Pattern Analysis & Machine Intelligence 43, 01 (Jan. 2021), 172–186. DOI: 10.1109/TPAMI.2019.2929257

João Carreira and Andrew Zisserman. 2017. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4724–4733. DOI: 10.1109/CVPR.2017.502

Victor Costa, Leandro Tavares, Ruhan Conceição, Luciano Agostini, Brenda Santana, Tatiana Lebedeff, and Guilherme Corrêa. 2025. A Lightweight I3D-Based Approach for Real-Time Brazilian Sign Language Recognition. In Proceedings of the 31st Brazilian Symposium on Multimedia and the Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 501–505. DOI: 10.5753/webmedia.2025.16072

David Vinicius da Silva, Valter Estevam, and David Menotti. 2024. Less is More: Concatenating Videos for Sign Language Translation from a Small Set of Signs. In 2024 37th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). 1–6. DOI: 10.1109/SIBGRAPI62404.2024.10716311

Rodrigo Zempulski Fanucchi, Arlindo Rodrigues Galvão Junior, Gabriel da Mata Marques, Lucas Brandão Rodrigues, Anderson da Silva Soares, and Telma Woerle Lima Soares. 2024. Fine-Tuning a Video Masked Autoencoder to Develop an Augmented Reality Application for Brazilian Sign Language Interpretation. In Proceedings of the 26th Symposium on Virtual and Augmented Reality. 275–278.

Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023).

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770–778. DOI: 10.1109/CVPR.2016.90

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics Human Action Video Dataset. arXiv:1705.06950 [cs.CV] [link]

Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, et al. 2019. Mediapipe: A framework for perceiving and processing reality. In Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR), Vol. 2019. Long Beach, CA, 2.

Ministério da Educação do Brasil. 2023. Dia Nacional do Surdo destaca importância da inclusão e da acessibilidade. [link]. Estimates that about 5% of the Brazilian population has some form of hearing impairment..

Presidência da República do Brasil. 2002. Lei nº 10.436, de 24 de abril de 2002. [link]. Recognizes the Brazilian Sign Language (Libras) as a legal means of communication and expression..

Presidência da República do Brasil. 2005. Decreto nº 5.626, de 22 de dezembro de 2005. [link]. Regulates Lei nº 10.436/2002 and mandates the provision of accessibility for deaf people in education, media, and public services..

Ronice Müller de Quadros and Lodenir Becker Karnopp. 2007. Língua de sinais brasileira: estudos lingüísticos. (2007).

Tamires Martins Rezende. 2021. Reconhecimento automático de sinais da Libras: desenvolvimento da base de dados MINDS-Libras e modelos de redes convolucionais. Ph. D. Dissertation. Universidade Federal de Minas Gerais.

Tamires Martins Rezende, Sílvia Grasiella Moreira Almeida, and Frederico Gadelha Guimarães. 2021. Development and validation of a Brazilian sign language database for human gesture recognition. Neural Computing and Applications 33, 16 (2021), 10449–10467.

Ailton José Rodrigues. 2021. V-LIBRASIL: uma base de dados com sinais na língua brasileira de sinais (Libras). Master’s thesis. Universidade Federal de Pernambuco, Recife.

Fida Mohammad Thoker and Juergen Gall. 2019. Cross-modal knowledge distillation for action recognition. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 6–10.

Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems 35 (2022), 10078–10093.