Automatic Spoken-to-Sign Language Translation: A Review of Key Technologies, Limitations, and Interdisciplinary Opportunities
Resumo
The communication barrier between deaf and hearing individuals remains a global challenge in academic, professional, and social contexts. Although numerous countries officially recognize their respective national sign languages, the availability of automatic, real-time, and linguistically accurate translation systems from spoken into sign languages remains limited in both coverage and quality. This paper reviews the key technologies, limitations, and interdisciplinary opportunities on computational approaches for real-time spoken-to-sign language translation. Following a systematic search of IEEE Xplore, Scopus, ACM Digital Library, and SciELO databases conducted in accordance with PRISMA guidelines, covering research published between 2020 and 2024, and applying predefined inclusion and exclusion criteria, fifteen peer-reviewed studies were identified. The analysis of key enabling technologies, including automatic speech recognition (ASR), natural language processing (NLP), 3D avatar animation, and emerging applications in virtual reality (VR), identified major linguistic and technical challenges, such as the representation of non-manual signals, multimodal grammar processing, and the scarcity of high-quality datasets. Results revealed four dominant technological approaches: ASR–NLP pipelines, avatar-based rendering systems, VR/AR learning tools, and sensor-based gesture recognition platforms. Despite substantial advances, major limitations include low representation of non-manual signals, restricted vocabulary coverage, and high hardware dependency. The review concludes that interdisciplinary research integrating linguistics, AI, and human-computer interaction is essential for developing culturally adaptive and accessible solutions for the deaf community.
Palavras-chave:
sign language translation, virtual reality, 3D avatars, automatic speech recognition
Referências
W. Silva and M. Oliveira. Automatic speech recognition for Libras translation. Brazilian Journal of Assistive Technologies, 5(2), 45–60, 2021.
R. Sousa and F. Lima. Convolutional neural networks adaptation for regional accents in Libras translation. Brazilian AI Conference Proceedings, 2023.
A. Nogueira and J. Pereira. 3D avatar for automatic Libras translation. Graphics Computing Symposium, 2020.
World Federation of the Deaf. World Federation of the Deaf, 2021. Available: [link]
VLibras Project. Tools for automatic translation from Portuguese to Libras, 2020. Available: [link]
SignAll Inc. Computer vision technology for deaf-hearing communication, 2022. Available: [link]
J. W. Creswell. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications, 2017.
B. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering. Keele University and Durham University Joint Report, 2007.
M. J. Page et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Systematic Reviews, 10, 1–11, 2021.
A. Booth, A. Sutton, M. Clowes, and M. Martyn-St James. Systematic Approaches to a Successful Literature Review, 2021.
R. Z. Fanucchi et al. Fine-Tuning a Video Masked Autoencoder to Develop an Augmented Reality Application for Brazilian Sign Language Interpretation. Anais do Simposio de Realidade Virtual e Aumentada (SVR), 2024.
D. R. Nazareth, M. A. Dos Santos Alencar, and J. F. De Magalhaes Netto. ELRA - Teaching Brazilian Sign Language Using Augmented Reality. Anais do Simposio de Realidade Virtual e Aumentada (SVR), 2014.
C. E. G. R. Alves, F. A. Boldt, and T. M. Paixao. Enhancing Brazilian Sign Language Recognition through Skeleton Image Representation. Anais da Conference on Graphics, Patterns and Images (SIBGRAPI), 2024.
D. V. da Silva, V. Estevam, and D. Menotti. Less is More: Concatenating Videos for Sign Language Translation from a Small Set of Signs. arXiv preprint arXiv:2409.01506, 2024.
L. de S. Arcanjo et al. Automatic Time-aware Recognition of Brazilian Sign Language Based on Dynamic Time Warping. Anais do Brazilian Symposium on Multimedia and the Web (WebMedia), 2024.
F. de A. Freitas et al. Leveraging Sign Language Processing with Formal SignWriting and Deep Learning Architectures. Anais da Brazilian Conference on Intelligent Systems (BRACIS), 2023.
H. Fang, T. Kwon, S. Bhardwaj, J. Zhang, Y. Shi, and S. Fei. DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation. Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys), 81–94, 2018.
N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden. Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10023–10033, 2020.
M. S. Alam, F. Islam, R. Rahman, and M. Rahman. ASL Champ!: A Virtual Reality Game with Deep-Learning Driven Sign Recognition. Applied Sciences, 13(1), 89, 2023.
H. Vaitkevicius, A. Blazauskas, A. Damasevicius, R. Maskeliunas, and R. Damasevicius. Recognition of American Sign Language Gestures in a Virtual Reality Using Leap Motion. Applied Sciences, 9(3), 445, 2019.
A. Lakhfif. Design and Implementation of a Virtual 3D Educational Environment to Improve Deaf Education. International Journal of Emerging Technologies in Learning (iJET), 15(12), 4–15, 2020.
N. Zirzow. Signing Avatars: Using Virtual Reality to Support Students with Hearing Loss. Rural Special Education Quarterly, 34(3), 27–32, 2015.
Y. Yuan, M. Zhao, and X. Xu. Translation Quality Evaluation of Sign Language Avatar. Proceedings of the 3rd Conference on Chinese Computational Linguistics (CCL), 567–578, 2024.
L. T. Nguyen, Q. V. Nguyen, and M. H. Le. Automatic Generation of a 3D Sign Language Avatar on AR Glasses Given 2D Videos of Human Signers. Proceedings of the 1st Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), 71–81, 2021.
A. Nunes, J. Silva, and R. Almeida. Innovation in Learning – The Use of Avatar for Brazilian Sign Language. Technical Report, Federal Institute of Technology, 2020.
K. Papadopoulos and G. D. Kollias. 3D Realistic Animation of Greek Sign Language’s Fingerspelled Signs. Computer Animation and Virtual Worlds, 31(6), 2020.
S. Sharma and R. Mehra. Virtual Indian Sign Language Interpreter Using 3D Avatars. Journal of Accessibility Technologies, 15(2), 112–120, 2021.
L. Quandt, S. Reinicke, and T. Muller. Signing Avatars in a New Dimension: Challenges and Opportunities in Virtual Reality. Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology (SLTAT), 2022.
A. Alethary, A. Aliwy, and N. S. Ali. Automated Arabic-Arabic Sign Language Translation System Based on 3D Avatar Technology. International Journal of Advances in Applied Sciences, 11(4), 383–396, 2022.
WIRED. This Startup Has Created AI-Powered Signing Avatars for the Deaf. WIRED Magazine, 2025. Available: [link]
A. Vaswani et al. Attention Is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), 5998–6008, 2017.
D. Chakladar, P. Kumar, S. Mandal, P. P. Roy, M. Iwamura, and B.-G. Kim. 3D Avatar Approach for Continuous Sign Movement Using Speech/Text. Applied Sciences, 11(8), 3439, 2021.
L. G. G. Almeida, N. V. de Vasconcelos, I. Winkler, and M. F. Catapan. Innovating Industrial Training with Immersive Metaverses: A Method for Developing Cross-Platform Virtual Reality Environments. Applied Sciences, 13(15), 8915, 2023. Available: [link]
G. Brito, D. Scheltes, A. Durval, J. Aragao, S. Santos, C. Lima, M. Soussa, and I. Winkler. Developing an University Immersive Campus: Guidelines for Beginner Developers. Anais do XXVI Simposio de Realidade Virtual e Aumentada, 261–265, 2024. Available: [link]
R. Sousa and F. Lima. Convolutional neural networks adaptation for regional accents in Libras translation. Brazilian AI Conference Proceedings, 2023.
A. Nogueira and J. Pereira. 3D avatar for automatic Libras translation. Graphics Computing Symposium, 2020.
World Federation of the Deaf. World Federation of the Deaf, 2021. Available: [link]
VLibras Project. Tools for automatic translation from Portuguese to Libras, 2020. Available: [link]
SignAll Inc. Computer vision technology for deaf-hearing communication, 2022. Available: [link]
J. W. Creswell. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications, 2017.
B. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering. Keele University and Durham University Joint Report, 2007.
M. J. Page et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Systematic Reviews, 10, 1–11, 2021.
A. Booth, A. Sutton, M. Clowes, and M. Martyn-St James. Systematic Approaches to a Successful Literature Review, 2021.
R. Z. Fanucchi et al. Fine-Tuning a Video Masked Autoencoder to Develop an Augmented Reality Application for Brazilian Sign Language Interpretation. Anais do Simposio de Realidade Virtual e Aumentada (SVR), 2024.
D. R. Nazareth, M. A. Dos Santos Alencar, and J. F. De Magalhaes Netto. ELRA - Teaching Brazilian Sign Language Using Augmented Reality. Anais do Simposio de Realidade Virtual e Aumentada (SVR), 2014.
C. E. G. R. Alves, F. A. Boldt, and T. M. Paixao. Enhancing Brazilian Sign Language Recognition through Skeleton Image Representation. Anais da Conference on Graphics, Patterns and Images (SIBGRAPI), 2024.
D. V. da Silva, V. Estevam, and D. Menotti. Less is More: Concatenating Videos for Sign Language Translation from a Small Set of Signs. arXiv preprint arXiv:2409.01506, 2024.
L. de S. Arcanjo et al. Automatic Time-aware Recognition of Brazilian Sign Language Based on Dynamic Time Warping. Anais do Brazilian Symposium on Multimedia and the Web (WebMedia), 2024.
F. de A. Freitas et al. Leveraging Sign Language Processing with Formal SignWriting and Deep Learning Architectures. Anais da Brazilian Conference on Intelligent Systems (BRACIS), 2023.
H. Fang, T. Kwon, S. Bhardwaj, J. Zhang, Y. Shi, and S. Fei. DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation. Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys), 81–94, 2018.
N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden. Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10023–10033, 2020.
M. S. Alam, F. Islam, R. Rahman, and M. Rahman. ASL Champ!: A Virtual Reality Game with Deep-Learning Driven Sign Recognition. Applied Sciences, 13(1), 89, 2023.
H. Vaitkevicius, A. Blazauskas, A. Damasevicius, R. Maskeliunas, and R. Damasevicius. Recognition of American Sign Language Gestures in a Virtual Reality Using Leap Motion. Applied Sciences, 9(3), 445, 2019.
A. Lakhfif. Design and Implementation of a Virtual 3D Educational Environment to Improve Deaf Education. International Journal of Emerging Technologies in Learning (iJET), 15(12), 4–15, 2020.
N. Zirzow. Signing Avatars: Using Virtual Reality to Support Students with Hearing Loss. Rural Special Education Quarterly, 34(3), 27–32, 2015.
Y. Yuan, M. Zhao, and X. Xu. Translation Quality Evaluation of Sign Language Avatar. Proceedings of the 3rd Conference on Chinese Computational Linguistics (CCL), 567–578, 2024.
L. T. Nguyen, Q. V. Nguyen, and M. H. Le. Automatic Generation of a 3D Sign Language Avatar on AR Glasses Given 2D Videos of Human Signers. Proceedings of the 1st Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), 71–81, 2021.
A. Nunes, J. Silva, and R. Almeida. Innovation in Learning – The Use of Avatar for Brazilian Sign Language. Technical Report, Federal Institute of Technology, 2020.
K. Papadopoulos and G. D. Kollias. 3D Realistic Animation of Greek Sign Language’s Fingerspelled Signs. Computer Animation and Virtual Worlds, 31(6), 2020.
S. Sharma and R. Mehra. Virtual Indian Sign Language Interpreter Using 3D Avatars. Journal of Accessibility Technologies, 15(2), 112–120, 2021.
L. Quandt, S. Reinicke, and T. Muller. Signing Avatars in a New Dimension: Challenges and Opportunities in Virtual Reality. Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology (SLTAT), 2022.
A. Alethary, A. Aliwy, and N. S. Ali. Automated Arabic-Arabic Sign Language Translation System Based on 3D Avatar Technology. International Journal of Advances in Applied Sciences, 11(4), 383–396, 2022.
WIRED. This Startup Has Created AI-Powered Signing Avatars for the Deaf. WIRED Magazine, 2025. Available: [link]
A. Vaswani et al. Attention Is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), 5998–6008, 2017.
D. Chakladar, P. Kumar, S. Mandal, P. P. Roy, M. Iwamura, and B.-G. Kim. 3D Avatar Approach for Continuous Sign Movement Using Speech/Text. Applied Sciences, 11(8), 3439, 2021.
L. G. G. Almeida, N. V. de Vasconcelos, I. Winkler, and M. F. Catapan. Innovating Industrial Training with Immersive Metaverses: A Method for Developing Cross-Platform Virtual Reality Environments. Applied Sciences, 13(15), 8915, 2023. Available: [link]
G. Brito, D. Scheltes, A. Durval, J. Aragao, S. Santos, C. Lima, M. Soussa, and I. Winkler. Developing an University Immersive Campus: Guidelines for Beginner Developers. Anais do XXVI Simposio de Realidade Virtual e Aumentada, 261–265, 2024. Available: [link]
Publicado
30/09/2025
Como Citar
FERREIRA, William; TANAJURA, Pedro; SIMAS, João Pedro; FERREIRA, Yasmim Thasla Santos; SOUSSA, Márcio; WINKLER, Ingrid.
Automatic Spoken-to-Sign Language Translation: A Review of Key Technologies, Limitations, and Interdisciplinary Opportunities. In: SIMPÓSIO DE REALIDADE VIRTUAL E AUMENTADA (SVR), 27. , 2025, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 494-503.
